in reply to No More Meaningless Benchmarks!

Heh, I did the same thing chromatic did and created another benchmark, then figured I should really get work done instead on futzing around on Perlmonks. If I think the speed difference between quoting operators is the limiting part of my program, I really have too much time on my hands or a program that doesn't do anything interesting (or is something Perl shouldn't be doing). Any result from Benchmark that is less than about 7% shouldn't be trusted (see links later in post).

Another problem with this particular benchmark is that the interesting part of the code (the quote thing) is tied up in the I/O and other bits. The numbers don't show the difference between the quotes, they show the difference between the various other things combined with it. You have to pull that stuff apart to figure out what's going on. One way to do that is just what chromatic did: can you get the same answer twice?

I went through a lot of this discussion in Wasting time thinking about wasted time, in my Benchmarking Perl" talk at Uniforum last year, and in my Benchmarking chapter for Mastering Perl. Benchmarks are for the people who constantly ask themselves "How can this be wrong" and do the thinking and the work to find every hole in what they are doing. You can't stop when you have a number that you think is right. You have to do more benchmarks in other ways to see if you can use what you know to make predictions that hold up to reality. :)

--
brian d foy <brian@stonehenge.com>
Subscribe to The Perl Review