Skip Menu |
 

This queue is for tickets about the Test-Most CPAN distribution.

Report information
The Basics
Id: 104465
Status: open
Priority: 0/
Queue: Test-Most

People
Owner: Nobody in particular
Requestors: sebastian_rose [...] gmx.de
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Severe Bug: tests run after bail_on_fail
Date: Fri, 15 May 2015 22:37:17 +0200
To: bug-Test-Most [...] rt.cpan.org
From: Sebastian Rose <sebastian_rose [...] gmx.de>
Download (untitled) / with headers
text/plain 225b
Hello maintainers, I just (2015-05-15) installed Test::Most. I was especially interested in the handy error handling features. Unfortunately I found, what I consider a severe bug (on the first glance). A few examples:
Download (untitled) / with headers
text/x-perl 281b
use Test::Most 'no_plan'; die_on_fail; ok 1, 'Throw on fail'; restore_fail; # The error happens with and without this line. bail_on_fail; ok 0, 'BAIL_OUT on fail'; # Expected to stop testing immediately! restore_fail; ok 1, 'Dangerous test';
Download (untitled) / with headers
text/plain 726b
sh# prove t/test_most_error_handling.t -v 01 t/test_most_error_handling.t .. 02 ok 1 - Throw on fail 03 not ok 2 - BAIL_OUT on fail 04 ok 3 - Dangerous test - WILL BE EXECUTED 05 1..3 06 Bailout called. Further testing stopped: Test failed. BAIL OUT!. 07 Bail out! Test failed. BAIL OUT!. 08 09 # Failed test 'BAIL_OUT on fail' 10 # at t/test_most_error_handling.t line 9. 11 # Looks like you failed 1 test of 3. 12 FAILED--Further testing stopped: Test failed. BAIL OUT!. NOTE lines 4 and 11! 3 Tests were executed - instead of 2. According to the docs, this does not seem to be the intended behaviour (to me). Here comes the behaviour that I would expect (triggering BAIL_OUT "by hand"):
Download (untitled) / with headers
text/x-perl 194b
use Test::Most 'no_plan'; die_on_fail; ok 1, 'Throw on fail'; bail_on_fail; ok( 0, 'BAIL_OUT on fail' ) or BAIL_OUT(" S T O P "); restore_fail; ok 1, 'Dangerous test - WILL NOT BE EXECUTED';
Download (untitled) / with headers
text/plain 592b
sh# prove t/test_most_error_handling.t -v 01 t/test_most_error_handling.t .. 02 ok 1 - Throw on fail 03 not ok 2 - BAIL_OUT on fail 04 Bailout called. Further testing stopped: S T O P 05 Bail out! S T O P 06 07 # Failed test 'BAIL_OUT on fail' 08 # at t/test_most_error_handling.t line 7. 09 FAILED--Further testing stopped: S T O P Only two tests executed. BAIL_OUT stopped the testing immediatly. 'Dangerous test' is NOT executed. -- Ich setzte einen Fuß in die Luft, und sie trug. (Hilde Domin)
Download (untitled) / with headers
text/plain 363b
On 2015-05-15 13:37:34, sebastian_rose@gmx.de wrote: Show quoted text
> Hello maintainers, > > > I just (2015-05-15) installed Test::Most. I was especially interested > in the handy error handling features. > > Unfortunately I found, what I consider a severe bug (on the first > glance). A few examples: > >
What is your installed version of Test::More and Test::Builder?
Download (untitled) / with headers
text/plain 772b
I seem to recall a discussion about this, and a Test::Builder feature request that never got fleshed out. Basically if you bail-out as soon as the failure happens you get no debugging, you simply get the 'not ok' line and no info on what line it failed. So the way this feature is often implemented is to note that you should bail, but let stuff continue so that diag gets printed, then bail the next time ok is called. So you have a choice between immediate bailing with no helpful diag, or helpful diag, but a delayed bail. Here are related discussions, I may have some details wrong, so if the conversation contradicts anything above believe the conversation: https://github.com/Test-More/test-more/issues/373 https://github.com/Test-More/test-more/wiki/Stop-On-Fail
Subject: Re: [rt.cpan.org #104465] Severe Bug: tests run after bail_on_fail
Date: Fri, 15 May 2015 23:43:06 +0200
To: "Karen Etheridge via RT" <bug-Test-Most [...] rt.cpan.org>
From: Sebastian Rose <sebastian_rose [...] gmx.de>
Download (untitled) / with headers
text/plain 892b
"Karen Etheridge via RT" <bug-Test-Most@rt.cpan.org> writes: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=104465 > > > On 2015-05-15 13:37:34, sebastian_rose@gmx.de wrote:
>> Hello maintainers, >> >> >> I just (2015-05-15) installed Test::Most. I was especially interested >> in the handy error handling features. >> >> Unfortunately I found, what I consider a severe bug (on the first >> glance). A few examples: >> >>
> > > What is your installed version of Test::More and Test::Builder? >
Hm - sorry - the info was missing. I updated all my Perl modules just before I tried Test::Most. $Test::More::VERSION = 1.001014 $Test::Builder::VERSION = 1.001014 This is perl 5, version 20, subversion 1 (v5.20.1) built for x86_64-linux-thread-multi -- Ich setzte einen Fuß in die Luft, und sie trug. (Hilde Domin)
Subject: Re: [rt.cpan.org #104465] Severe Bug: tests run after bail_on_fail
Date: Fri, 15 May 2015 23:49:21 +0200
To: "Chad Granum via RT" <bug-Test-Most [...] rt.cpan.org>
From: Sebastian Rose <sebastian_rose [...] gmx.de>
"Chad Granum via RT" <bug-Test-Most@rt.cpan.org> writes: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=104465 > > > I seem to recall a discussion about this, and a Test::Builder feature request > that never got fleshed out. Basically if you bail-out as soon as the failure > happens you get no debugging, you simply get the 'not ok' line and no info on > what line it failed. So the way this feature is often implemented is to note > that you should bail, but let stuff continue so that diag gets printed, then > bail the next time ok is called. > > So you have a choice between immediate bailing with no helpful diag, or helpful diag, but a delayed bail. > > Here are related discussions, I may have some details wrong, so if the conversation contradicts anything above believe the conversation: > > https://github.com/Test-More/test-more/issues/373 > https://github.com/Test-More/test-more/wiki/Stop-On-Fail
Hallo Chad, the problem is, that the docs say BAIL_OUT is called. I would expect the tests to stop. "bail_on_fail" bail_on_fail; is_deeply $foo, bar, '... we bail out if this fails'; This function, if called, will cause the test suite to BAIL_OUT() if any tests fail after it. ... If the "DIE_ON_FAIL" or "BAIL_ON_FAIL" environment variables are true, any tests which use "Test::Most" will throw an exception or call BAIL_OUT on test failure. perldoc Test::More BAIL_OUT BAIL_OUT($reason); Indicates to the harness that things are going so badly all testing should terminate. This includes the running of any additional test scripts. This is typically used when testing cannot continue such as a critical module failing to compile or a necessary external utility not being available such as a database connection failing. The test will exit with 255. For even better control look at Test::Most. Debugging can be done differently (and why not add more tests to find an error?). -- Ich setzte einen Fuß in die Luft, und sie trug.
Download (untitled) / with headers
text/plain 484b
Test::Most can document/promise whatever it wants, that doesn't change the technical issues it would have to overcome to do what it promises while still using Test::Builder under the hood. I do not maintain Test::Most (thank God), but whoever does needs to see how Test::Most implements die on fail and either fix it as a bug, or document its behavior better. I just wanted to make sure everyone involved knew about the Test::Builder architectural issues that had bearing on this bug.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.