in reply to No More Meaningless Benchmarks!

At what point does a benchmark become meaningless?

Whilst I agree that whether you use single quotes or double quotes is a total non-issue in performance terms, at what point did the benchmark become meaningless? It could have been at any of the following points in time:

  1. At the point when the question was asked?

    Maybe we know the answer (it doesn't matter), instinctively, but the OP did cite (wrongly or not) an authoritative sounding source: "One of the O'Reilly Perl books says ...".

    Given that there are some 42 O'Reilly Perl books currently listed, how many people could instinctively say that this advice doesn't appear in one of them?

    And who's gonna say that given a sufficiently well constructed benchmark, testing the right things, a difference couldn't be legitimately detected under some particular circumstances--albeit that the differences would be so small as to be meaningless in any realistic context?

  2. At the point the benchmarker decide to do it?

    Whilst it may be obvious to those of us that have used Perl for a while that such differences are (mostly) meaningless in the bigger context on most occasions, the benchmarker was responding to the OP's question and attempting to give definitive feedback.

    That seems legitimate to me?

  3. At the point that the benchmarker decided he had found a difference worthy of publishing?

    Given the the OP's question(*) and (albeit hearsay) cited reference that there might be a difference, attempting to answer it is surely legitimate.

    * Is any question meaningless, if the asker simply does not know the answer?

    And not knowing enough to either construct a meaningful benchmark, or correctly interpret the results can't be a crime can it? A few people have made that mistake, including some fairly experienced ones.

    So, whilst the benchmark itself, and the interpretation of what it showed where both incorrect, is it correct to characterise the benchmark itself as "meaningless"?

I'd say that the benchmark in question was not meaningless if for no other reason than it caused you to post this meditation. By doing so you've done (at least) the following good things:

But for these benefits to have ensued, someone had to ask the question; someone had to construct and run the benchmark; and someone had to publish the (misinterpreted) results.

And for each of the two person involved in creating the circumstances that led to your meditation, there are probably 10 or 20 or 100 hundred more that will read your meditation; that would have never asked, never have benchmarked, or never published what they found; and so would have gone on believing that quoting style really was a performance issue.

Still think it meaningless?


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.