Skip Menu |
 

Preferred bug tracker

Please visit the preferred bug tracker to report your issue.

This queue is for tickets about the CPAN-Reporter CPAN distribution.

Report information
The Basics
Id: 30084
Status: rejected
Priority: 0/
Queue: CPAN-Reporter

People
Owner: Nobody in particular
Requestors: SREZIC [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: (no value)
Fixed in: 1.02



Subject: Test failure reported as PASS
Download (untitled) / with headers
text/plain 1.2k
Just seen while testing IO::Tee on a MSWin32 system: the test has one test failure, but CPAN::Reporter thinks that the tests passes. See here for a similar report (mine, which uses CPAN::Reporter 1.02, is not yet on the cpantesters site, as it seems): http://www.nntp.perl.org/group/perl.cpan.testers/2007/09/msg617415.html I am not sure whose fault this is. The test file in IO::Tee does not use any of the Test:: modules, but just does "ok" and "not ok" printing itself. Therefore no special exit code is set (like it is done when using Test::*), but I can't remember that setting the exit code was required in the old days. Maybe it's ExtUtils::MakeMaker fault because it does not run the test file through Test::Harness, as it seems. Here a sample test rule generated by ExtUtils::MakeMaker on a Unix system with a single test.pl file: test_dynamic :: pure_all PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-I$(INST_LIB)" -I$(INST_ARCHLIB)" $(TEST_FILE) And here the test rule when using t/*.t: test_dynamic :: pure_all PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-MExtUtils::Command::MM" "-e" "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES) Only the latter seems to use the Test::Harness. Regards, Slaven
Download (untitled) / with headers
text/plain 980b
Thank you for reporting this. However, this is a problem with IO::Tee, not CPAN::Reporter. IO::Tee provides only a single test.pl file. ExtUtils::MakeMaker runs that by itself -- not through Test::Harness. (By contrast, Module::Build runs even test.pl files through the harness -- which might be a "bug" since it's a big change from EU::MM. This is actually why there is t/54_test_report_split.t test in CPAN::Reporter -- for cases where EU:MM and M::B give different answers on a distribution.) In this case, test.pl finishes with an exit code of 0. From the perspective of an external program, that's success. It can print whatever it wants, ok, not ok, hello world, etc. The only thing that matters is the exit code. If it happened to use Test::More, Test::More has an END block that overrides the exit value to signal whether tests passed or not. In this case, IO::Tee writes TAP "by hand" and doesn't set an exit code appropriately. Thus -- it's an IO::Tee bug.
Subject: Re: [rt.cpan.org #30084] Test failure reported as PASS
Date: 18 Oct 2007 21:41:38 +0200
To: bug-CPAN-Reporter [...] rt.cpan.org
From: Slaven Rezic <slaven [...] rezic.de>
Download (untitled) / with headers
text/plain 2.3k
" via RT" <bug-CPAN-Reporter@rt.cpan.org> writes: Show quoted text
> <URL: http://rt.cpan.org/Ticket/Display.html?id=30084 > > > Thank you for reporting this. > > However, this is a problem with IO::Tee, not CPAN::Reporter. IO::Tee > provides only a single test.pl file. ExtUtils::MakeMaker runs that by > itself -- not through Test::Harness. > > (By contrast, Module::Build runs even test.pl files through the harness > -- which might be a "bug" since it's a big change from EU::MM. This is > actually why there is t/54_test_report_split.t test in CPAN::Reporter -- > for cases where EU:MM and M::B give different answers on a distribution.) > > In this case, test.pl finishes with an exit code of 0. From the > perspective of an external program, that's success. It can print > whatever it wants, ok, not ok, hello world, etc. The only thing that > matters is the exit code. > > If it happened to use Test::More, Test::More has an END block that > overrides the exit value to signal whether tests passed or not. In this > case, IO::Tee writes TAP "by hand" and doesn't set an exit code > appropriately. Thus -- it's an IO::Tee bug. >
I am not sure. If I create a module stub using h2xs from an old perl distribution (say, 5.00505), then it creates only a test.pl file and it looks like this: __BEGIN__ # Before `make install' is performed this script should be runnable with # `make test'. After `make install' it should work as `perl test.pl' ######################### We start with some black magic to print on failure. # Change 1..1 below to 1..last_test_to_print . # (It may become useful if the test is moved to ./t subdirectory.) BEGIN { $| = 1; print "1..1\n"; } END {print "not ok 1\n" unless $loaded;} use Foo; $loaded = 1; print "ok 1\n"; ######################### End of black magic. # Insert your test code below (better if it prints "ok 13" # (correspondingly "not ok 13") depending on the success of chunk 13 # of the test code): __END__ No "exit 1" in case of errors here. I also cannot remember ever to put an "exit ..." line into my hand-written test files (before Test::* module were available). So my opinion is that EUMM should run the test.pl also through Test::Harness. Regards, Slaven -- Slaven Rezic - slaven <at> rezic <dot> de babybike - routeplanner for cyclists in Berlin handheld (e.g. Compaq iPAQ with Linux) version of bbbike http://bbbike.sourceforge.net
From: DAGOLDEN [...] cpan.org
Download (untitled) / with headers
text/plain 1.2k
On Thu Oct 18 15:44:18 2007, slaven@rezic.de wrote: Show quoted text
> No "exit 1" in case of errors here. I also cannot remember ever to put > an "exit ..." line into my hand-written test files (before Test::* > module were available). So my opinion is that EUMM should run the > test.pl also through Test::Harness.
That's an EU::MM issue, not a CPAN::Reporter issue. Here is the result of installing this on MSWin32 with CPAN::Reporter turned *off*: [snip] ok 25 not ok 26 ok 27 KENSHAN/IO-Tee-0.64.tar.gz C:\strawberry-perl\dmake\bin\dmake.EXE test -- OK Running make install Prepending C:\Temp\cpan\build\IO-Tee-0.64-Ya0DLi/blib/arch C:\Temp\cpan\build\IO -Tee-0.64-Ya0DLi/blib/lib to PERL5LIB for 'install' Installing C:\strawberry-perl\perl\site\lib\IO\Tee.pm Writing c:\strawberry-perl\perl\site\lib\auto\IO\Tee\.packlist Appending installation info to c:\strawberry-perl\perl\lib/perllocal.pod KENSHAN/IO-Tee-0.64.tar.gz C:\strawberry-perl\dmake\bin\dmake.EXE install UNINST=1 -- OK As you can see, CPAN treats the test as a success and proceeds to installation. As that is all that CPAN::Reporter is trying to assess, then the current behavior is "correct" -- even if the current behavior isn't ideal. I'm going to re-close this issue. David


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.