Lies, Damn Lies and Benchmarksby liz (Monsignor)
|on Mar 21, 2004 at 22:27 UTC||Need Help??|
Sometimes you feel like you need a slap to the head.
Yesterday was such a day when I finally figured out that a strange phenomenon, which has been discussed at one Perl Mongers meeting, as well as in a BOF on a Perl Workshop, is actually caused by a very bad benchmark. Lies and Damn Lies, indeed.
The phenomenon was that a benchmark showed that it was faster to do:
This seems counter-intuitive, but the benchmark* was repeatable and showed that using a shift and a seperate assignment of the hash was almost three times as fast as doing in one list assignment. Just to make sure, I ran the benchmark with both 5.8.3 and 5.6.2. The benchmark and the result:
While preparing a some articles about micro-optimizations for Perl Monks, I decided to test this some more. Because I suddenly realised that I was testing this without parameters actually being passed. So I figured I'd do a run with parameters actually being passed. And everything changed. Observe:
Huh? What's this? So doing it in one list assignment apparently is more efficient if you pass enough parameters to the subroutine. Hmmm... but what if we don't have parameters for the hash assignment. Surely then it would be faster to use the approach using shift()? Nope.
The list assignment was still more efficient. Huh? Had I been testing wrong. Ok, surely without any parameters passed, it would be faster to use shift()? Again, nope!
So what was the difference between my original benchmark and this one, apart from the overhead of calling an extra subroutine for each iteration? What was I missing?
The thing I was missing was in this little piece of documentation in perlsub:
and indeed, if I changed the call from foo() to &foo, the following benchmark came about:
And indeed, that's a lot closer to the original benchmark.
What further conclusions can be drawn from this? Not sure, I guess I'll leave that as an excercise to the reader. ;-)
This just goes to show that you should always check, doublecheck and triplecheck your benchmarks.
*Please note that benchmarks can be off by 5 to 10% between runs. I've run the each benchmark multiple times, but some of them were run while running on battery power, and others were run when my iBook was plugged in. Within one benchmark, I always had the situation consistent, so the results between 5.6.2 and 5.8.3 of a benchmark can be compared (keeping in mind the 5 - 10% uncertainty for each run, of course).