Skip Menu |
 

Preferred bug tracker

Please visit the preferred bug tracker to report your issue.

This queue is for tickets about the Dumbbench CPAN distribution.

Report information
The Basics
Id: 81157
Status: new
Priority: 0/
Queue: Dumbbench

People
Owner: Nobody in particular
Requestors: ribasushi [...] leporine.io
Cc:
AdminCc:

Bug Information
Severity: Unimportant
Broken in: (no value)
Fixed in: (no value)



Subject: Inconsistent results of identical benchmark - what am I missing?
Download (untitled) / with headers
text/plain 2.6k
I opted to file an RT instead of contacting you by email, so that the conversation is archived for future generations. I am attaching a standalone script which attempts to test two identical copies of a handmade accessor (defined on line 37) subjecting it to 200 cycles of 5 get/set operations (line 64). I tweaked/tightened the Dumbbench options to the best of my understanding (lines 12-28). The machine is silent (no active processes) and the CPU is in a low-power/no speed variations/no turbo state. Since you use Time::HiRes::time() calls everywhere (awesome!), I was able to replace your wallclock measurements with cputime measurements, by monkeypatching Time::HiRes (lines 210-213). With all this I get rather large differences in my resutls. While one could have attributed this to noise, it does not seem to be the case as the *difference* is reproducible. The benchmark starts randomly placing either of the two on top. But further iterations will *always* have the former "winner" on top, every time for the rest of the run. I've ran hundreds if not thousands of benches, and I can't seem to find the culprit. Note - I am not questioning the 2 percentage-points error in the results - I understand this is somewhat expected. I am questioning why there is no random component to this error - why does it always fall a certain reproducible way. See example below. Cheers rabbit@Ahasver:~/devel$ perl bench_wtf Perl 5.016002, take 1: Rate HANDMADE1 HANDMADE2 HANDMADE1 748.422+-0.074/s -- -1.4% HANDMADE2 759.414+-0.076/s 1.5% -- Perl 5.016002, take 2: Rate HANDMADE1 HANDMADE2 HANDMADE1 742.16+-0.074/s -- -1.4% HANDMADE2 752.902+-0.073/s 1.4% -- Perl 5.016002, take 3: Rate HANDMADE1 HANDMADE2 HANDMADE1 738.118+-0.073/s -- -1.3% HANDMADE2 747.668+-0.074/s 1.3% -- Perl 5.016002, take 4: Rate HANDMADE1 HANDMADE2 HANDMADE1 735.658+-0.073/s -- -1.4% HANDMADE2 745.831+-0.074/s 1.4% -- Perl 5.016002, take 5: Rate HANDMADE1 HANDMADE2 HANDMADE1 733.735+-0.073/s -- -1.7% HANDMADE2 746.405+-0.073/s 1.7% -- Perl 5.016002, take 6: Rate HANDMADE1 HANDMADE2 HANDMADE1 737.488+-0.071/s -- -0.9% HANDMADE2 744.129+-0.073/s 0.9% -- Perl 5.016002, take 7: Rate HANDMADE1 HANDMADE2 HANDMADE1 735.209+-0.071/s -- -1.7% HANDMADE2 748.179+-0.074/s 1.8% -- Perl 5.016002, take 8: Rate HANDMADE1 HANDMADE2 HANDMADE1 735.55+-0.072/s -- -1.4% HANDMADE2 746.062+-0.074/s 1.4% --
Subject: bench_wtf
Download bench_wtf
application/octet-stream 5.3k

Message body not shown because it is not plain text.



This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.