Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
Sometimes you feel like you need a slap to the head.

Yesterday was such a day when I finally figured out that a strange phenomenon, which has been discussed at one Perl Mongers meeting, as well as in a BOF on a Perl Workshop, is actually caused by a very bad benchmark. Lies and Damn Lies, indeed.

The phenomenon was that a benchmark showed that it was faster to do:

my $self = shift; my %param = @_;
rather than:
my ($self,%param) = @_;
This seems counter-intuitive, but the benchmark* was repeatable and showed that using a shift and a seperate assignment of the hash was almost three times as fast as doing in one list assignment. Just to make sure, I ran the benchmark with both 5.8.3 and 5.6.2. The benchmark and the result:
use Benchmark qw(cmpthese); cmpthese( -2,{ list => sub { my ($self,%param) = @_ }, shiftit => sub { my $self = shift; my %param = @_ }, } ); __END__ 5.8.3 Rate list shiftit list 78392/s -- -73% shiftit 292680/s 273% -- 5.6.2 Rate list shiftit list 94936/s -- -68% shiftit 297717/s 214% --

While preparing a some articles about micro-optimizations for Perl Monks, I decided to test this some more. Because I suddenly realised that I was testing this without parameters actually being passed. So I figured I'd do a run with parameters actually being passed. And everything changed. Observe:

sub list { my ($self,%param) = @_ } sub shiftit { my $self = shift; my %param = @_ } cmpthese( -2,{ list => sub { list( qw(foo bar baz) ) }, shiftit => sub { shiftit( qw(foo bar baz) ) }, } ); __END__ 5.8.3 Rate shiftit list shiftit 70621/s -- -11% list 79125/s 12% -- 5.6.2 Rate shiftit list shiftit 89341/s -- -10% list 98866/s 11% --
Huh? What's this? So doing it in one list assignment apparently is more efficient if you pass enough parameters to the subroutine. Hmmm... but what if we don't have parameters for the hash assignment. Surely then it would be faster to use the approach using shift()? Nope.
cmpthese( -2,{ list => sub { list( qw(foo) ) }, shiftit => sub { shiftit( qw(foo) ) }, } ); __END__ 5.8.3 Rate shiftit list shiftit 164545/s -- -19% list 204158/s 24% -- 5.6.2 Rate shiftit list shiftit 179408/s -- -12% list 203990/s 14% --
The list assignment was still more efficient. Huh? Had I been testing wrong. Ok, surely without any parameters passed, it would be faster to use shift()? Again, nope!
cmpthese( -2,{ list => sub { list() }, shiftit => sub { shiftit() }, } ); __END__ 5.8.3 Rate shiftit list shiftit 217350/s -- -12% list 247334/s 14% -- 5.6.2 Rate shiftit list shiftit 212031/s -- -14% list 246447/s 16% --
So what was the difference between my original benchmark and this one, apart from the overhead of calling an extra subroutine for each iteration? What was I missing?

The thing I was missing was in this little piece of documentation in perlsub:
To call subroutines: NAME(LIST); # & is optional with parentheses. NAME LIST; # Parentheses optional if predeclared/import +ed. &NAME(LIST); # Circumvent prototypes. &NAME; # Makes current @_ visible to called subrout +ine.
and indeed, if I changed the call from foo() to &foo, the following benchmark came about:
cmpthese( -2,{ list => sub { &list }, shiftit => sub { &shiftit }, } ); __END__ 5.8.3 Rate list shiftit list 126315/s -- -62% shiftit 333577/s 164% -- 5.6.2 Rate list shiftit list 135814/s -- -61% shiftit 343986/s 153% --
And indeed, that's a lot closer to the original benchmark.

What further conclusions can be drawn from this? Not sure, I guess I'll leave that as an excercise to the reader. ;-)

This just goes to show that you should always check, doublecheck and triplecheck your benchmarks.


*Please note that benchmarks can be off by 5 to 10% between runs. I've run the each benchmark multiple times, but some of them were run while running on battery power, and others were run when my iBook was plugged in. Within one benchmark, I always had the situation consistent, so the results between 5.6.2 and 5.8.3 of a benchmark can be compared (keeping in mind the 5 - 10% uncertainty for each run, of course).

In reply to Lies, Damn Lies and Benchmarks by liz

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others exploiting the Monastery: (6)
    As of 2018-01-24 02:57 GMT
    Find Nodes?
      Voting Booth?
      How did you see in the new year?

      Results (255 votes). Check out past polls.