Skip Menu |

This queue is for tickets about the Test-Harness CPAN distribution.

Report information
The Basics
Id: 2101
Status: resolved
Priority: 0/
Queue: Test-Harness

Owner: andy [...]
Requestors: frag [...]

Bug Information
Severity: Wishlist
Broken in: (no value)
Fixed in: (no value)


Subject: Randomizing tests
Add randomization of test order to Test::Harness. See attached file for some extended thoughts...
Download random-tests.txt
text/plain 3.3k
OK, you said this would be problematic, but I figured I should add it to rt anyway. (Plus, I wanted to play with rt.) So, I was going to just put here 'randomize order by which test scripts are invoked in runtests()'. But I started thinking (and this started getting long), and I'm wondering, regardless of practicality, if randomizing on the file order really makes sense. I've always been under the impression that each test script should undo any changes it's made. E.g., each script creates an object it needs to make, and destroys it by the time it's finished, rather than 01new.t creating an object, and 02set.t testing mutators on the object created by 01new.t. Or that you shouldn't have 01write.t create a file and 02read.t read from that same file. (Whether this is correct, you might want to touch on this briefly in your presentation, when you get to the logic of writing multiple scripts.) If my understanding of this isn't correct, and it's acceptable or desirable for each test script to depend on actions taken by previous test scripts, then the whole notion of randomizing the run order of the test scripts is a non-starter, at least for any situation where that sort of ordering is required. If my understanding is correct, and test scripts should leave no trace when they finish, then randomization at the level of scripts is pointless, since each test script will start out with essentially a tabula rasa. (Unless the test scripts themselves are broken and fail to undo their changes, or if they crash midway through, before getting the chance to undo changes.) So, what's needed is randomization at the level of the individual tests, within a single script. I've got no idea how to do this using the current situation where tests are just simple Perl scripts. (One way would be to read the file, and juggle all lines that execute tests, and eval the result; but this is obviously a dumb idea.) There's probably no way to do this at all so that you could just take a standard test script and arbitrarily toggle between random and sequential testing. You'd have to require the script writer to set up the script to be random from the get-go. Basically, the script writer would take each test to be randomized (along with any ancillary set-up or tear-down code that the test depends upon) and wrap the code into a variable. (I.e. as a sub, or a code reference, or as a string, or maybe as a labelled block, or possibly even within pod directives.) Then, at the end of the script, a randomization function would be invoked with the names/code references/labels/whatever of each test to be randomized. Tests in the body of the script that aren't wrapped up and set aside for randomization would be executed first, in sequence, as with any normal test script. The randomization function could first look at the script's input to see if a record of a previous run was passed in (either via STDIN or as a file to open) and if there is, replicate that order; otherwise by default it would generate a new random order. So, basically, we'd have to have a Test::Randomize module that would do a lot of this work, except for the bit where the test writer has to chop up the script into randomizable chunks and pass them into a run_randomly() function. But is this even something that's really needed? Is it worth the extra work?
This will be handled by prove, which I've started working on in the 2.31 series.
Prove in 2.32 lets you --shuffle the tests, but it does not yet allow playback of a series.
Download (untitled) / with headers
text/plain 127b
As of 3.04 this is possible: $ prove -b --shuffle t and then $ prove -b --state=last to replay. Thanks for the suggestion.

This service is sponsored and maintained by Best Practical Solutions and runs on infrastructure.

Please report any issues with to