Skip Menu |
 

This queue is for tickets about the Test-Class CPAN distribution.

Report information
The Basics
Id: 92553
Status: open
Priority: 0/
Queue: Test-Class

People
Owner: Nobody in particular
Requestors: ville.misaki [...] bluecieloecm.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: The order of tests run across multiple modules is not deterministic.
Date: Wed, 29 Jan 2014 09:20:15 +0000
To: "bug-Test-Class [...] rt.cpan.org" <bug-Test-Class [...] rt.cpan.org>
From: Ville Misaki <ville.misaki [...] bluecieloecm.com>
Download (untitled) / with headers
text/plain 744b
We are building a test suite with dependencies across the tests, so they must be run in strict order. Because of the large amounts of tests to be included, we have to split them into multiple modules, running the tests of each in order. The test methods within each module run in case-sensitive alphabetical order, but the modules themselves appear to be run in random order. Currently using the Test::Class version 0.41, and the following patch appears to work for us: 365c365 < TEST_OBJECT: foreach my $t (@tests) { --- Show quoted text
> TEST_OBJECT: foreach my $t (sort @tests) {
This change will effectively add a new step ("All of the modules in alphabetical order") before the first step on the documented section "RUNNING ORDER OF METHODS".
Download (untitled) / with headers
text/plain 1.5k
On Wed Jan 29 04:20:51 2014, ville.misaki@bluecieloecm.com wrote: Show quoted text
> We are building a test suite with dependencies across the tests, so > they must be run in strict order. Because of the large amounts of > tests to be included, we have to split them into multiple modules, > running the tests of each in order. The test methods within each > module run in case-sensitive alphabetical order, but the modules > themselves appear to be run in random order. > > Currently using the Test::Class version 0.41, and the following patch > appears to work for us: > > 365c365 > < TEST_OBJECT: foreach my $t (@tests) { > ---
> > TEST_OBJECT: foreach my $t (sort @tests) {
> > This change will effectively add a new step ("All of the modules in > alphabetical order") before the first step on the documented section > "RUNNING ORDER OF METHODS".
Speaking as someone who (a) does not claim to be an expert on Test::Class but (b) does use Test::Class on $job ... I do not think the change requested above should be implemented. In most smoke testing setups that I am aware of, you should not want "a test suite with dependencies across the tests", as that would prevent you from making use of parallelism in testing, e.g., 'make -j8 test'. The fact that the various test classes *can* be run in an indeterminate order is, IMO, a feature, not a bug. If the original posters want to sub-class Test::Class for that purpose, then more power to them. But I do not think we should change Test::Class's behavior for this purpose. Thank you very much. Jim Keenan
Subject: RE: [rt.cpan.org #92553] The order of tests run across multiple modules is not deterministic.
Date: Tue, 8 Jul 2014 06:10:18 +0000
To: "bug-Test-Class [...] rt.cpan.org" <bug-Test-Class [...] rt.cpan.org>
From: Ville Misaki <ville.misaki [...] bluecieloecm.com>
Download (untitled) / with headers
text/plain 2.7k
Show quoted text
> > We are building a test suite with dependencies across the tests, so > > they must be run in strict order. Because of the large amounts of > > tests to be included, we have to split them into multiple modules, > > running the tests of each in order. The test methods within each > > module run in case-sensitive alphabetical order, but the modules > > themselves appear to be run in random order. > > > > Currently using the Test::Class version 0.41, and the following patch > > appears to work for us: > > > > 365c365 > > < TEST_OBJECT: foreach my $t (@tests) { > > ---
> > > TEST_OBJECT: foreach my $t (sort @tests) {
> > > > This change will effectively add a new step ("All of the modules in > > alphabetical order") before the first step on the documented section > > "RUNNING ORDER OF METHODS".
> > Speaking as someone who (a) does not claim to be an expert on > Test::Class but (b) does use Test::Class on $job ... > > I do not think the change requested above should be implemented. > In most smoke testing setups that I am aware of, you should not want > "a test suite with dependencies across the tests", as that would > prevent you from making use of parallelism in testing, e.g., > 'make -j8 test'. The fact that the various test classes *can* be > run in an indeterminate order is, IMO, a feature, not a bug. > > If the original posters want to sub-class Test::Class for that > purpose, then more power to them. But I do not think we should > change Test::Class's behavior for this purpose.
I don't consider this as a bug per se, but rather a lack of documentation and/or a deviance from the behavior elsewhere in the module. The running order of the methods is clearly defined within the test classes. The test classes can also be run in order if they are provided as parameters to runtests(); which would actually break with my originally suggested patch, but putting the sort around _test_classes() should be safe: < @tests = _test_classes( $base_class ); --- Show quoted text
> @tests = sort _test_classes( $base_class );
Normally we indeed do not want any dependencies between tests, and only ran into this when creating tests for setting up the configuration of our system -- the configuration objects have references to each other, requiring certain objects to be created before others. We have also already worked around this case by creating our own objects establishing the order of tests, and are currently using an unpatched vanilla version of Test::Class. An alternative fix for this issue would be just to document this behavior, i.e. deterministic order when test classes are given as parameters and nondeterministic otherwise. I will leave it to the maintainers to decide which approach would be more suitable.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.