Skip Menu |
 

This queue is for tickets about the Test-Harness CPAN distribution.

Report information
The Basics
Id: 27377
Status: resolved
Priority: 0/
Queue: Test-Harness

People
Owner: andy [...] hexten.net
Requestors: doughera [...] lafayette.edu
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



CC: Perl Porters <perl5-porters [...] perl.org>
Subject: Counting bug in Test::Harness with unexpected early exit
Date: Fri, 1 Jun 2007 16:20:56 -0400 (EDT)
To: bug-test-harness [...] rt.cpan.org
From: Andy Dougherty <doughera [...] lafayette.edu>
Download (untitled) / with headers
text/plain 2.9k
Test::Harness gives wrong numbers when the test script dies unexpectedly. This happens for both bleadperl and perl-5.8.8. Consider the following simple test file bug.t: use Test; BEGIN { plan tests => 6 } ok(1); ok(0); ok(0); ok(0); die("unexpected crash!"); ok(5); ok(6); Running ./perl -Ilib utils/prove bug.t produces the incorrect result: bug....NOK 2/6# Failed test 2 in bug.t at line 4 # bug.t line 4 is: ok(0); bug....NOK 3/6# Failed test 3 in bug.t at line 5 # bug.t line 5 is: ok(0); bug....NOK 4/6# Failed test 4 in bug.t at line 6 # bug.t line 6 is: ok(0); unexpected crash! at bug.t line 7. bug....dubious Test returned status 255 (wstat 65280, 0xff00) DIED. FAILED tests 2-6 Failed 5/6 tests, 16.67% okay Failed Test Stat Wstat Total Fail List of Failed ------------------------------------------------------------------------------- bug.t 255 65280 6 7 2-6 Failed 1/1 test scripts. 5/6 subtests failed. Files=1, Tests=6, 0 wallclock secs ( 0.01 cusr + 0.00 csys = 0.01 CPU) Failed 1/1 test programs. 5/6 subtests failed. Note in the table that it claims there are 6 tests (correct) but that 7 failed. However, in the summary, it correctly notes that 5 out of the 6 tests failed. perl-5.8.8 fails in a similar way, but with the added incorrect percentage of 116.67% failing. Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- bug.t 255 65280 6 7 116.67% 2-6 I speculate that it's double-counting all tests after the die(). For an extreme example, consider the output I get from running 'make test' for parrot: Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- t/compilers/pge/p5regex/p5rx.t 0 3 960 1466 152.71% 228-960 t/compilers/pge/p6regex/01-regex. 0 3 ?? ?? % ?? t/examples/shootout.t 1 256 20 1 5.00% 16 t/pmc/complex.t 5 1280 53 5 9.43% 32 42-43 46-47 t/src/hash.t 1 256 11 1 9.09% 6 t/src/io.t 255 65280 20 18 90.00% 12-20 (4 subtests UNEXPECTEDLY SUCCEEDED), 11 tests and 277 subtests skipped. Failed 6/304 test scripts, 98.03% okay. 449/6669 subtests failed, 93.27% okay. where the numbers listed as faling for p5rx.t and io.t are exactly twice the number you get from the "List of Failed". Note too that there is no obvious way to get the number '449' in the summary from any of the numbers in the table above! -- Andy Dougherty doughera@lafayette.edu
Fixed in 2.99, thanks.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.