Skip Menu |
 

This queue is for tickets about the Test-Class CPAN distribution.

Report information
The Basics
Id: 92553
Status: open
Priority: 0/
Queue: Test-Class

People
Owner: Nobody in particular
Requestors: ville.misaki [...] bluecieloecm.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009001)(6009001)(199002)(189002)(79102001)(77982001)(59766001)(65816001)(31966008)(19300405004)(56776001)(87266001)(50986001)(15975445006)(2656002)(15202345003)(47976001)(85306002)(47446002)(74662001)(80022001)(81816001)(54316002)(74502001)(81686001)(80976001)(76482001)(90146001)(74366001)(19580395003)(83072002)(33646001)(85852003)(74876001)(54356001)(81342001)(63696002)(74316001)(74706001)(56816005)(4396001)(83322001)(47736001)(16236675002)(49866001)(92566001)(81542001)(76176001)(93136001)(76786001)(69226001)(51856001)(93516002)(76796001)(53806001)(46102001)(86362001)(94316002)(76576001)(87936001)(3826001)(24736002);DIR:OUT;SFP:1101;SCL:1;SRVR:DB3PR05MB298;H:DB3PR05MB299.eurprd05.prod.outlook.com;CLIP:2001:16f8:24:3:852e:e15f:1931:7df0;FPR:;InfoNoRecordsA:1;MX:1;LANG:en;
From ville.misaki [...] bluecieloecm.com Wed Jan 29 04: 20:50 2014
MIME-Version: 1.0
X-Spam-Status: No, score=-2.6 tagged_above=-99.9 required=10 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001] autolearn=ham
X-Spam-Flag: NO
X-Virus-Checked: Checked
Content-Language: en-US
Content-Type: multipart/alternative; boundary="_000_18100bdfce3d4898afe5abfb850d9fb5DB3PR05MB299eurprd05pro_"
Message-ID: <18100bdfce3d4898afe5abfb850d9fb5 [...] DB3PR05MB299.eurprd05.prod.outlook.com>
X-Virus-Scanned: Debian amavisd-new at bestpractical.com
X-MS-Tnef-Correlator:
X-Spam-Score: -2.6
Received: from localhost (localhost [127.0.0.1]) by hipster.bestpractical.com (Postfix) with ESMTP id E96EA240E6E for <cpan-bug+Test-Class [...] hipster.bestpractical.com>; Wed, 29 Jan 2014 04:20:49 -0500 (EST)
Received: from hipster.bestpractical.com ([127.0.0.1]) by localhost (hipster.bestpractical.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W-iFBP6rrTht for <cpan-bug+Test-Class [...] hipster.bestpractical.com>; Wed, 29 Jan 2014 04:20:28 -0500 (EST)
Received: from la.mx.develooper.com (x1.develooper.com [207.171.7.70]) by hipster.bestpractical.com (Postfix) with SMTP id 8F724240494 for <bug-Test-Class [...] rt.cpan.org>; Wed, 29 Jan 2014 04:20:27 -0500 (EST)
Received: (qmail 16501 invoked by alias); 29 Jan 2014 09:20:25 -0000
Received: from mail-db3lp0078.outbound.protection.outlook.com (HELO emea01-db3-obe.outbound.protection.outlook.com) (213.199.154.78) by la.mx.develooper.com (qpsmtpd/0.28) with ESMTP; Wed, 29 Jan 2014 01:20:21 -0800
Received: from DB3PR05MB299.eurprd05.prod.outlook.com (10.141.6.20) by DB3PR05MB298.eurprd05.prod.outlook.com (10.141.6.18) with Microsoft SMTP Server (TLS) id 15.0.859.15; Wed, 29 Jan 2014 09:20:15 +0000
Received: from DB3PR05MB299.eurprd05.prod.outlook.com ([10.141.6.20]) by DB3PR05MB299.eurprd05.prod.outlook.com ([10.141.6.20]) with mapi id 15.00.0859.020; Wed, 29 Jan 2014 09:20:15 +0000
Delivered-To: cpan-bug+Test-Class [...] hipster.bestpractical.com
Subject: The order of tests run across multiple modules is not deterministic.
Return-Path: <ville.misaki [...] bluecieloecm.com>
Thread-Index: Ac8c0lDRwoDX3frySOSZvDq7nT1zkA==
X-RT-Mail-Extension: test-class
X-Original-To: cpan-bug+Test-Class [...] hipster.bestpractical.com
X-Spam-Check-BY: la.mx.develooper.com
X-Forefront-PRVS: 01068D0A20
Date: Wed, 29 Jan 2014 09:20:15 +0000
X-Originatororg: bluecieloecm.com
X-Spam-Level:
X-MS-Has-Attach:
Thread-Topic: The order of tests run across multiple modules is not deterministic.
X-Originating-Ip: [2001:16f8:24:3:852e:e15f:1931:7df0]
Accept-Language: en-US
To: "bug-Test-Class [...] rt.cpan.org" <bug-Test-Class [...] rt.cpan.org>
From: Ville Misaki <ville.misaki [...] bluecieloecm.com>
X-RT-Interface: Email
Content-Length: 0
content-type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
X-RT-Original-Encoding: ascii
Content-Length: 744
Download (untitled) / with headers
text/plain 744b
We are building a test suite with dependencies across the tests, so they must be run in strict order. Because of the large amounts of tests to be included, we have to split them into multiple modules, running the tests of each in order. The test methods within each module run in case-sensitive alphabetical order, but the modules themselves appear to be run in random order. Currently using the Test::Class version 0.41, and the following patch appears to work for us: 365c365 < TEST_OBJECT: foreach my $t (@tests) { --- Show quoted text
> TEST_OBJECT: foreach my $t (sort @tests) {
This change will effectively add a new step ("All of the modules in alphabetical order") before the first step on the documented section "RUNNING ORDER OF METHODS".
content-type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
X-RT-Original-Encoding: ascii
Content-Length: 7048
MIME-Version: 1.0
In-Reply-To: <18100bdfce3d4898afe5abfb850d9fb5 [...] DB3PR05MB299.eurprd05.prod.outlook.com>
X-Mailer: MIME-tools 5.504 (Entity 5.504)
Content-Disposition: inline
X-RT-Interface: Web
References: <18100bdfce3d4898afe5abfb850d9fb5 [...] DB3PR05MB299.eurprd05.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
Message-ID: <rt-4.0.18-21516-1404780368-1037.92553-0-0 [...] rt.cpan.org>
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
X-RT-Encrypt: 0
X-RT-Sign: 0
Content-Length: 1555
Download (untitled) / with headers
text/plain 1.5k
On Wed Jan 29 04:20:51 2014, ville.misaki@bluecieloecm.com wrote: Show quoted text
> We are building a test suite with dependencies across the tests, so > they must be run in strict order. Because of the large amounts of > tests to be included, we have to split them into multiple modules, > running the tests of each in order. The test methods within each > module run in case-sensitive alphabetical order, but the modules > themselves appear to be run in random order. > > Currently using the Test::Class version 0.41, and the following patch > appears to work for us: > > 365c365 > < TEST_OBJECT: foreach my $t (@tests) { > ---
> > TEST_OBJECT: foreach my $t (sort @tests) {
> > This change will effectively add a new step ("All of the modules in > alphabetical order") before the first step on the documented section > "RUNNING ORDER OF METHODS".
Speaking as someone who (a) does not claim to be an expert on Test::Class but (b) does use Test::Class on $job ... I do not think the change requested above should be implemented. In most smoke testing setups that I am aware of, you should not want "a test suite with dependencies across the tests", as that would prevent you from making use of parallelism in testing, e.g., 'make -j8 test'. The fact that the various test classes *can* be run in an indeterminate order is, IMO, a feature, not a bug. If the original posters want to sub-class Test::Class for that purpose, then more power to them. But I do not think we should change Test::Class's behavior for this purpose. Thank you very much. Jim Keenan
MIME-Version: 1.0
X-Spam-Flag: NO
X-Virus-Scanned: Debian amavisd-new at bestpractical.com
content-type: text/plain; charset="utf-8"
X-RT-Original-Encoding: utf-8
X-Spam-Score: -4.601
Received: from localhost (localhost [127.0.0.1]) by hipster.bestpractical.com (Postfix) with ESMTP id 7658624081D for <cpan-bug+Test-Class [...] hipster.bestpractical.com>; Tue, 8 Jul 2014 02:10:32 -0400 (EDT)
Received: from hipster.bestpractical.com ([127.0.0.1]) by localhost (hipster.bestpractical.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o8Dyp2WSNaSa for <cpan-bug+Test-Class [...] hipster.bestpractical.com>; Tue, 8 Jul 2014 02:10:30 -0400 (EDT)
Received: from la.mx.develooper.com (x1.develooper.com [207.171.7.70]) by hipster.bestpractical.com (Postfix) with SMTP id B05BF2407DA for <bug-Test-Class [...] rt.cpan.org>; Tue, 8 Jul 2014 02:10:30 -0400 (EDT)
Received: (qmail 22962 invoked by alias); 8 Jul 2014 06:10:29 -0000
Received: from mail-am1lp0011.outbound.protection.outlook.com (HELO emea01-am1-obe.outbound.protection.outlook.com) (213.199.154.11) by la.mx.develooper.com (qpsmtpd/0.28) with ESMTP; Mon, 07 Jul 2014 23:10:24 -0700
Received: from DB3PR05MB299.eurprd05.prod.outlook.com (10.141.6.20) by DB3PR05MB297.eurprd05.prod.outlook.com (10.141.6.13) with Microsoft SMTP Server (TLS) id 15.0.980.8; Tue, 8 Jul 2014 06:10:19 +0000
Received: from DB3PR05MB299.eurprd05.prod.outlook.com ([10.141.6.20]) by DB3PR05MB299.eurprd05.prod.outlook.com ([10.141.6.20]) with mapi id 15.00.0980.000; Tue, 8 Jul 2014 06:10:19 +0000
Delivered-To: cpan-bug+Test-Class [...] hipster.bestpractical.com
Subject: RE: [rt.cpan.org #92553] The order of tests run across multiple modules is not deterministic.
Thread-Index: Ac8c0lDRwoDX3frySOSZvDq7nT1zkB9c7E+AAAptBNA=
X-Spam-Check-BY: la.mx.develooper.com
Date: Tue, 8 Jul 2014 06:10:18 +0000
X-Originatororg: bluecieloecm.com
X-Spam-Level:
To: "bug-Test-Class [...] rt.cpan.org" <bug-Test-Class [...] rt.cpan.org>
X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:
Content-Transfer-Encoding: base64
X-Forefront-Antispam-Report: SFV:NSPM;SFS:(6009001)(51704005)(199002)(189002)(54356999)(21056001)(101416001)(106356001)(83322001)(76176999)(107046002)(2351001)(83072002)(85852003)(33646001)(105586002)(66066001)(2656002)(64706001)(87936001)(74662001)(46102001)(85306003)(110136001)(76576001)(76482001)(20776003)(92566001)(95666004)(86362001)(74316001)(77982001)(50986999)(4396001)(99396002)(31966008)(79102001)(74502001)(81542001)(107886001)(80022001)(81342001)(108616002)(24736002);DIR:OUT;SFP:;SCL:1;SRVR:DB3PR05MB297;H:DB3PR05MB299.eurprd05.prod.outlook.com;FPR:;MLV:sfv;PTR:InfoNoRecords;MX:1;LANG:en;
In-Reply-To: <rt-4.0.18-21516-1404780368-1639.92553-6-0 [...] rt.cpan.org>
X-Spam-Status: No, score=-4.601 tagged_above=-99.9 required=10 tests=[BAYES_00=-1.9, FROM_OUR_RT=-2, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001] autolearn=ham
X-RT-Interface: API
Content-Language: en-US
References: <RT-Ticket-92553 [...] rt.cpan.org> <18100bdfce3d4898afe5abfb850d9fb5 [...] DB3PR05MB299.eurprd05.prod.outlook.com> <rt-4.0.18-21516-1404780368-1639.92553-6-0 [...] rt.cpan.org>
Message-ID: <e88d1b302adb478a86b2dd971f4a21d5 [...] DB3PR05MB299.eurprd05.prod.outlook.com>
X-MS-Tnef-Correlator:
Return-Path: <ville.misaki [...] bluecieloecm.com>
X-RT-Mail-Extension: test-class
X-Original-To: cpan-bug+Test-Class [...] hipster.bestpractical.com
X-Forefront-PRVS: 0266491E90
X-MS-Has-Attach:
Thread-Topic: [rt.cpan.org #92553] The order of tests run across multiple modules is not deterministic.
X-Originating-Ip: [83.98.170.254]
Accept-Language: en-US
From: Ville Misaki <ville.misaki [...] bluecieloecm.com>
RT-Message-ID: <rt-4.0.18-19684-1404799833-1354.92553-0-0 [...] rt.cpan.org>
Content-Length: 2788
Download (untitled) / with headers
text/plain 2.7k
Show quoted text
> > We are building a test suite with dependencies across the tests, so > > they must be run in strict order. Because of the large amounts of > > tests to be included, we have to split them into multiple modules, > > running the tests of each in order. The test methods within each > > module run in case-sensitive alphabetical order, but the modules > > themselves appear to be run in random order. > > > > Currently using the Test::Class version 0.41, and the following patch > > appears to work for us: > > > > 365c365 > > < TEST_OBJECT: foreach my $t (@tests) { > > ---
> > > TEST_OBJECT: foreach my $t (sort @tests) {
> > > > This change will effectively add a new step ("All of the modules in > > alphabetical order") before the first step on the documented section > > "RUNNING ORDER OF METHODS".
> > Speaking as someone who (a) does not claim to be an expert on > Test::Class but (b) does use Test::Class on $job ... > > I do not think the change requested above should be implemented. > In most smoke testing setups that I am aware of, you should not want > "a test suite with dependencies across the tests", as that would > prevent you from making use of parallelism in testing, e.g., > 'make -j8 test'. The fact that the various test classes *can* be > run in an indeterminate order is, IMO, a feature, not a bug. > > If the original posters want to sub-class Test::Class for that > purpose, then more power to them. But I do not think we should > change Test::Class's behavior for this purpose.
I don't consider this as a bug per se, but rather a lack of documentation and/or a deviance from the behavior elsewhere in the module. The running order of the methods is clearly defined within the test classes. The test classes can also be run in order if they are provided as parameters to runtests(); which would actually break with my originally suggested patch, but putting the sort around _test_classes() should be safe: < @tests = _test_classes( $base_class ); --- Show quoted text
> @tests = sort _test_classes( $base_class );
Normally we indeed do not want any dependencies between tests, and only ran into this when creating tests for setting up the configuration of our system -- the configuration objects have references to each other, requiring certain objects to be created before others. We have also already worked around this case by creating our own objects establishing the order of tests, and are currently using an unpatched vanilla version of Test::Class. An alternative fix for this issue would be just to document this behavior, i.e. deterministic order when test classes are given as parameters and nondeterministic otherwise. I will leave it to the maintainers to decide which approach would be more suitable.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.