Re: No More Meaningless Benchmarks!

At what point does a benchmark become meaningless?

Whilst I agree that whether you use single quotes or double quotes is a total non-issue in performance terms, at what point did the benchmark become meaningless? It could have been at any of the following points in time:

1. At the point when the question was asked?

   Maybe we know the answer (it doesn't matter), instinctively, but the OP did cite (wrongly or not) an authoritative sounding source: "One of the O'Reilly Perl books says ...".

   Given that there are some 42 O'Reilly Perl books currently listed, how many people could instinctively say that this advice doesn't appear in one of them?

   And who's gonna say that given a sufficiently well constructed benchmark, testing the right things, a difference couldn't be legitimately detected under some particular circumstances--albeit that the differences would be so small as to be meaningless in any realistic context?

2. At the point the benchmarker decide to do it?

   Whilst it may be obvious to those of us that have used Perl for a while that such differences are (mostly) meaningless in the bigger context on most occasions, the benchmarker was responding to the OP's question and attempting to give definitive feedback.

   That seems legitimate to me?

3. At the point that the benchmarker decided he had found a difference worthy of publishing?

   Given the the OP's question(*) and (albeit hearsay) cited reference that there might be a difference, attempting to answer it is surely legitimate.

   * Is any question meaningless, if the asker simply does not know the answer?

   And not knowing enough to either construct a meaningful benchmark, or correctly interpret the results can't be a crime can it? A few people have made that mistake, including some fairly experienced ones.

   So, whilst the benchmark itself, and the interpretation of what it showed where both incorrect, is it correct to characterise the benchmark itself as "meaningless"?

I'd say that the benchmark in question was not meaningless if for no other reason than it caused you to post this meditation. By doing so you've done (at least) the following good things:

• You informed the OP of the question to not take seemingly authoritative information at face value. Just because you read it in a book, even if the book is from a respected publisher in the field, it doesn't mean it is correct.

  So, how to verify it? You ask a question or you benchmark.

• You informed the benchmarker that constructing good benchmarks is hard. And interpreting the results you get is perhaps even harder.

  Perhaps most importantly, that when you benchmark, and get results that fly in the face of common opinion, question them.

• You brought all these things to the attention of the wider audience, some of whom may have also read the same statements in the same book, or possibly heard them through hearsay.

  I'm working on the assumption that the OP's quote and reference are correct, but even if they are not, the idea that utilising single quotes has a performance advantage, is real enough that it needs to be questioned and dispelled.

But for these benefits to have ensued, someone had to ask the question; someone had to construct and run the benchmark; and someone had to publish the (misinterpreted) results.

And for each of the two person involved in creating the circumstances that led to your meditation, there are probably 10 or 20 or 100 hundred more that will read your meditation; that would have never asked, never have benchmarked, or never published what they found; and so would have gone on believing that quoting style really was a performance issue.

Still think it meaningless?