Skip Menu |
 

This queue is for tickets about the Perl-Dist-Strawberry CPAN distribution.

Report information
The Basics
Id: 58545
Status: open
Priority: 0/
Queue: Perl-Dist-Strawberry

People
Owner: Nobody in particular
Requestors: adamk [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Strawberry needs a post-build QA test process
Download (untitled) / with headers
text/plain 1.1k
Over the last 6 months or so, most of the serious bugs that have been reported relate to mistakes that we really should have picked up in beta testing, but didn't. Examples: Bad PPM repository, badly broken C stuff, distributions missing support DLLs etc. I think we've seen enough of these that we should consider this to be a systematic flaw in the Strawberry production process rather than a collection of small oversights. I'd like to propose we develop a post-installation QA mechanism, so that beta testers and other can drop the installer onto a machine, fire up CPAN and just "install Perl::Dist::Strawberry::QA" or something similar. These post-production tests would potentially require that the test machines have MySQL, Postgres etc databases installed on them, and have Test::Database configured. It might try some basic tests for MySQL and Postgres (not necesarily the entire test suite), SQLite, installation via PPM, etc (installing the QA module itself tests CPAN installation). We could add a series of different regression tests to this module as we uncover and fix various problems which aren't caught by the module's own test scripts.
Download (untitled) / with headers
text/plain 2.1k
On Sun Jun 20 22:41:01 2010, ADAMK wrote: Show quoted text
> Over the last 6 months or so, most of the serious bugs that have been > reported relate to mistakes that we really should have picked up in beta > testing, but didn't. > > Examples: Bad PPM repository, badly broken C stuff, distributions > missing support DLLs etc. > > I think we've seen enough of these that we should consider this to be a > systematic flaw in the Strawberry production process rather than a > collection of small oversights. > > I'd like to propose we develop a post-installation QA mechanism, so that > beta testers and other can drop the installer onto a machine, fire up > CPAN and just "install Perl::Dist::Strawberry::QA" or something similar. > > These post-production tests would potentially require that the test > machines have MySQL, Postgres etc databases installed on them, and have > Test::Database configured. > > It might try some basic tests for MySQL and Postgres (not necesarily the > entire test suite), SQLite, installation via PPM, etc (installing the QA > module itself tests CPAN installation). > > We could add a series of different regression tests to this module as we > uncover and fix various problems which aren't caught by the module's own > test scripts.
Personally, I agree. I think instead of a CPAN dist, we need to make it a script that we can wrap with PAR::Packer that would run on a clean system. (Because we can't really QA some things well with a working Strawberry already installed, is my thought.) Fun first problem (and this would catch a few different types of problems): A post-build check for missing files between the .zip and .msi. It's easy enough to code... you just could not start with Strawberry already installed. (We still need to be able to QA-test 5.10, which still has hard-coding.) Strawberry would have to be extracted from the .zip, a File::List::Object object created, and then deleted. Then we'd install Strawberry from the .msi into the same directory, grab another FLO object created, and uninstall, and then we'd do along the lines of this script: http://cpansearch.perl.org/src/CSJEWELL/File-List-Object-0.202/eg/filecheck.pl Maybe fail if the count is not 0 for either one, or something. That'd be test 1.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.