Yesterday was such a day when I finally figured out that a strange phenomenon, which has been discussed at one Perl Mongers meeting, as well as in a BOF on a Perl Workshop, is actually caused by a very bad benchmark. Lies and Damn Lies, indeed.
The phenomenon was that a benchmark showed that it was faster to do:
rather than:my $self = shift; my %param = @_;
This seems counter-intuitive, but the benchmark* was repeatable and showed that using a shift and a seperate assignment of the hash was almost three times as fast as doing in one list assignment. Just to make sure, I ran the benchmark with both 5.8.3 and 5.6.2. The benchmark and the result:my ($self,%param) = @_;
use Benchmark qw(cmpthese); cmpthese( -2,{ list => sub { my ($self,%param) = @_ }, shiftit => sub { my $self = shift; my %param = @_ }, } ); __END__ 5.8.3 Rate list shiftit list 78392/s -- -73% shiftit 292680/s 273% -- 5.6.2 Rate list shiftit list 94936/s -- -68% shiftit 297717/s 214% --
While preparing a some articles about micro-optimizations for Perl Monks, I decided to test this some more. Because I suddenly realised that I was testing this without parameters actually being passed. So I figured I'd do a run with parameters actually being passed. And everything changed. Observe:
Huh? What's this? So doing it in one list assignment apparently is more efficient if you pass enough parameters to the subroutine. Hmmm... but what if we don't have parameters for the hash assignment. Surely then it would be faster to use the approach using shift()? Nope.sub list { my ($self,%param) = @_ } sub shiftit { my $self = shift; my %param = @_ } cmpthese( -2,{ list => sub { list( qw(foo bar baz) ) }, shiftit => sub { shiftit( qw(foo bar baz) ) }, } ); __END__ 5.8.3 Rate shiftit list shiftit 70621/s -- -11% list 79125/s 12% -- 5.6.2 Rate shiftit list shiftit 89341/s -- -10% list 98866/s 11% --
The list assignment was still more efficient. Huh? Had I been testing wrong. Ok, surely without any parameters passed, it would be faster to use shift()? Again, nope!cmpthese( -2,{ list => sub { list( qw(foo) ) }, shiftit => sub { shiftit( qw(foo) ) }, } ); __END__ 5.8.3 Rate shiftit list shiftit 164545/s -- -19% list 204158/s 24% -- 5.6.2 Rate shiftit list shiftit 179408/s -- -12% list 203990/s 14% --
So what was the difference between my original benchmark and this one, apart from the overhead of calling an extra subroutine for each iteration? What was I missing?cmpthese( -2,{ list => sub { list() }, shiftit => sub { shiftit() }, } ); __END__ 5.8.3 Rate shiftit list shiftit 217350/s -- -12% list 247334/s 14% -- 5.6.2 Rate shiftit list shiftit 212031/s -- -14% list 246447/s 16% --
and indeed, if I changed the call from foo() to &foo, the following benchmark came about:To call subroutines: NAME(LIST); # & is optional with parentheses. NAME LIST; # Parentheses optional if predeclared/import +ed. &NAME(LIST); # Circumvent prototypes. &NAME; # Makes current @_ visible to called subrout +ine.
And indeed, that's a lot closer to the original benchmark.cmpthese( -2,{ list => sub { &list }, shiftit => sub { &shiftit }, } ); __END__ 5.8.3 Rate list shiftit list 126315/s -- -62% shiftit 333577/s 164% -- 5.6.2 Rate list shiftit list 135814/s -- -61% shiftit 343986/s 153% --
What further conclusions can be drawn from this? Not sure, I guess I'll leave that as an excercise to the reader. ;-)
This just goes to show that you should always check, doublecheck and triplecheck your benchmarks.
Liz
*Please note that benchmarks can be off by 5 to 10% between runs. I've run the each benchmark multiple times, but some of them were run while running on battery power, and others were run when my iBook was plugged in. Within one benchmark, I always had the situation consistent, so the results between 5.6.2 and 5.8.3 of a benchmark can be compared (keeping in mind the 5 - 10% uncertainty for each run, of course).