Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

No More Meaningless Benchmarks!

by chromatic (Archbishop)
on Dec 08, 2006 at 21:24 UTC ( #588703=perlmeditation: print w/replies, xml ) Need Help??

In Use of single quotes versus double quotes, the original poster asks "Which is faster, single quotes or double quotes?"

As you might expect, a couple of people responded with "It doesn't matter!" grinder followed up with more detail, explaining exactly why it doesn't matter to Perl. In this case, Perl compiles a non-interpolating double-quoted string and a single-quoted string (which never interpolates) into the exact same opcodes.

That is, Perl does the same work.

As you might expect, another monk tried to be helpful by pulling out Benchmark. Re: Use of single quotes versus double quotes argues that, on one machine, single-quoted constant non-interpolating strings are 16% slower than double-quoted strings.

Yet that's completely meaningless. Why? Because it has nothing to do with Perl! As grinder demonstrated, perl performs the same actions for each type of string. The differences are elsewhere in the system. It could be that one subroutine has ops all within one memory page and the other has less locality. Looking at the length of the strings, I suspect that there's a buffer size issue as well (IO screws up all benchmarks). Who knows if there are other processes running?

For fun, I started with a modified benchmark that runs for only five seconds and uses warn to send all output to STDERR, so I could redirect it to /dev/null. Here's my start:

#!/usr/bin/perl use strict; use warnings; use Benchmark qw(cmpthese); my $tests = -5; sub single { warn 'This is a test'; } sub double { warn "This is a test"; } cmpthese( $tests, { single => \&single, double => \&double } );

... and the output:

$ perl 2> /dev/null Rate single double single 476308/s -- -1% double 481758/s 1% --

What happened if I changed the order of subroutine declarations? That might change the memory layout:

#!/usr/bin/perl use strict; use warnings; use Benchmark qw(cmpthese); my $tests = -5; sub double { warn "This is a test"; } sub single { warn 'This is a test'; } cmpthese( $tests, { single => \&single, double => \&double } );

The output is:

$ perl 2> /dev/null Rate double single double 474650/s -- -2% single 483537/s 2% --

Would removing strict and warnings have a similar effect?

#!/usr/bin/perl use Benchmark qw(cmpthese); my $tests = -5; sub double { warn "This is a test"; } sub single { warn 'This is a test'; } cmpthese( $tests, { single => \&single, double => \&double } );


perl 2> /dev/null Rate single double single 491116/s -- -2% double 502236/s 2% --

How about moving the declarations further?

#!/usr/bin/perl use Benchmark qw(cmpthese); my $tests = -5; sub single { warn 'This is a test'; } cmpthese( $tests, { single => \&single, double => \&double } ); sub double { warn "This is a test"; }


$ perl 2> /dev/null Rate double single double 449265/s -- -7% single 481758/s 7% --

There's one more code movement to try:

#!/usr/bin/perl use Benchmark qw(cmpthese); my $tests = -5; cmpthese( $tests, { single => \&single, double => \&double } ); sub double { warn "This is a test"; } sub single { warn 'This is a test'; }

As expected, the results are unpredictable:

$ perl 2> /dev/null Rate single double single 459485/s -- -0% double 460340/s 0% --

There are two possible morals to this story. The first is that moving code around can make your programs much faster. A better moral is know when benchmarks are useless.

Replies are listed 'Best First'.
Re: No More Meaningless Benchmarks!
by davido (Archbishop) on Dec 09, 2006 at 02:36 UTC

    Mark Twain said "There are three types of lies; lies, damn lies, and statistics."

    Unfortunately the world is dumbed-down enough that people can easily be bamboozled by stats. Just reflect on advertising slogans and claims (a few contrived examples):

    • "50% more vitamins than..." (Yes, but if X is Coke, and Y is Coke with a droplet of lemon juice in it, is 50% more actually significant? No.
    • "Your risk of contracting (name your disease) is ten times greater if you eat pistachios." (Yes, but if the risk without pistachios is one in 50,000,000, who cares if your modified risk is 1:5,000,000? You're still more likely to die of a car wreck.)
    • "Indexing a hash is twice as slow as indexing an array." Yes, but both are O(1) operations, which means if you're trying to optimize you might just want to look at the greater algorithm in which you're performing these lookups. If you actually care about the time differential, you might need to ask yourself why, and how you might eliminate the question in the first place.

    What people often don't get is the concept of statistical significance. ...and relative significance (which is related). Teaching that is beyond the scope of this node. ;)


Re: No More Meaningless Benchmarks!
by brian_d_foy (Abbot) on Dec 09, 2006 at 05:13 UTC

    Heh, I did the same thing chromatic did and created another benchmark, then figured I should really get work done instead on futzing around on Perlmonks. If I think the speed difference between quoting operators is the limiting part of my program, I really have too much time on my hands or a program that doesn't do anything interesting (or is something Perl shouldn't be doing). Any result from Benchmark that is less than about 7% shouldn't be trusted (see links later in post).

    Another problem with this particular benchmark is that the interesting part of the code (the quote thing) is tied up in the I/O and other bits. The numbers don't show the difference between the quotes, they show the difference between the various other things combined with it. You have to pull that stuff apart to figure out what's going on. One way to do that is just what chromatic did: can you get the same answer twice?

    I went through a lot of this discussion in Wasting time thinking about wasted time, in my Benchmarking Perl" talk at Uniforum last year, and in my Benchmarking chapter for Mastering Perl. Benchmarks are for the people who constantly ask themselves "How can this be wrong" and do the thinking and the work to find every hole in what they are doing. You can't stop when you have a number that you think is right. You have to do more benchmarks in other ways to see if you can use what you know to make predictions that hold up to reality. :)

    brian d foy <>
    Subscribe to The Perl Review
Re: No More Meaningless Benchmarks!
by BrowserUk (Pope) on Dec 09, 2006 at 08:33 UTC

    At what point does a benchmark become meaningless?

    Whilst I agree that whether you use single quotes or double quotes is a total non-issue in performance terms, at what point did the benchmark become meaningless? It could have been at any of the following points in time:

    1. At the point when the question was asked?

      Maybe we know the answer (it doesn't matter), instinctively, but the OP did cite (wrongly or not) an authoritative sounding source: "One of the O'Reilly Perl books says ...".

      Given that there are some 42 O'Reilly Perl books currently listed, how many people could instinctively say that this advice doesn't appear in one of them?

      And who's gonna say that given a sufficiently well constructed benchmark, testing the right things, a difference couldn't be legitimately detected under some particular circumstances--albeit that the differences would be so small as to be meaningless in any realistic context?

    2. At the point the benchmarker decide to do it?

      Whilst it may be obvious to those of us that have used Perl for a while that such differences are (mostly) meaningless in the bigger context on most occasions, the benchmarker was responding to the OP's question and attempting to give definitive feedback.

      That seems legitimate to me?

    3. At the point that the benchmarker decided he had found a difference worthy of publishing?

      Given the the OP's question(*) and (albeit hearsay) cited reference that there might be a difference, attempting to answer it is surely legitimate.

      * Is any question meaningless, if the asker simply does not know the answer?

      And not knowing enough to either construct a meaningful benchmark, or correctly interpret the results can't be a crime can it? A few people have made that mistake, including some fairly experienced ones.

      So, whilst the benchmark itself, and the interpretation of what it showed where both incorrect, is it correct to characterise the benchmark itself as "meaningless"?

    I'd say that the benchmark in question was not meaningless if for no other reason than it caused you to post this meditation. By doing so you've done (at least) the following good things:

    • You informed the OP of the question to not take seemingly authoritative information at face value. Just because you read it in a book, even if the book is from a respected publisher in the field, it doesn't mean it is correct.

      So, how to verify it? You ask a question or you benchmark.

    • You informed the benchmarker that constructing good benchmarks is hard. And interpreting the results you get is perhaps even harder.

      Perhaps most importantly, that when you benchmark, and get results that fly in the face of common opinion, question them.

    • You brought all these things to the attention of the wider audience, some of whom may have also read the same statements in the same book, or possibly heard them through hearsay.

      I'm working on the assumption that the OP's quote and reference are correct, but even if they are not, the idea that utilising single quotes has a performance advantage, is real enough that it needs to be questioned and dispelled.

    But for these benefits to have ensued, someone had to ask the question; someone had to construct and run the benchmark; and someone had to publish the (misinterpreted) results.

    And for each of the two person involved in creating the circumstances that led to your meditation, there are probably 10 or 20 or 100 hundred more that will read your meditation; that would have never asked, never have benchmarked, or never published what they found; and so would have gone on believing that quoting style really was a performance issue.

    Still think it meaningless?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: No More Meaningless Benchmarks!
by perrin (Chancellor) on Dec 08, 2006 at 21:43 UTC
    I totally agree! Also, know when to measure wall-clock time instead of CPU time. I often see people measuring CPU time when trying to benchmark things that use a database. That's totally useless! I/O-heavy tasks can't be measure by CPU time in a useful way.
Re: No More Meaningless Benchmarks!
by grinder (Bishop) on Dec 08, 2006 at 22:41 UTC

    oooh! I can't wait to make my programs go faster.

    /me rushes off to shuffle some code around. A 7% increase you say? That sounds like some easy work!

    • another intruder with the mooring in the heart of the Perl

Re: No More Meaningless Benchmarks!
by toma (Vicar) on Dec 09, 2006 at 06:41 UTC
    You've rediscovered that moving around code is one way to optimize! I can imagine an optimizer that would take advantage of this, automatically permuting code to find more efficient variations. This is something like the approach used by ATLAS (Automatically Tuned Linear Algebra Software).

    ATLAS is smart about not trusting something like the Benchmark module. I think it measures the results with a wall clock and will not compile if the system is otherwise busy. It takes a while to build.

    This type of optimization is probably not a desirable way to mangle your perl source code except for fun, but it might be a great technique for optimizing something important like a regular expression engine. Of course it would not just move code around, the optimizer would explore the space of different buffer sizes, etc.

    You can find ATLAS at I have used ATLAS in my projects and it worked great!

    It should work perfectly the first time! - toma
Re: No More Meaningless Benchmarks!
by gam3 (Curate) on Dec 09, 2006 at 22:31 UTC
    To point out that this benchmark is even more useless: I would think that the major differences between double and single quotes are going to be at compile time and not run time, and this benchmark only times the runtime differences.
    -- gam3
    A picture is worth a thousand words, but takes 200K.
      Excellent point!  ++gam3.

      To prove it to myself ...

      #!/usr/bin/perl -w use strict; use warnings; my $d = "This is a double-quote string\n"; my $s = 'This is a single-quote (apostrophe) string' . "\n"; print $d; print $s; # Results ... BEGIN { $^W = 1; } use warnings; use strict 'refs'; my $d = "This is a double-quote string\n"; my $s = "This is a single-quote (apostrophe) string\n"; print $d; print $s; syntax OK

      Looks pretty convincing to me!


      Very true. I was going to mention that myself.

      Still i believe a benchmark always says something. It's like profiling; it is often important to think about using it. Even if the benchmark tells nothing in a case like this one, still, the effort of thinking about why it does not tell anything is a useful exercise.

      hth --stephan
Re: No More Meaningless Benchmarks!
by geekphilosopher (Friar) on Dec 08, 2006 at 23:23 UTC

    Hah, that's hilarious. I thought much the same thing when I saw the benchmark after having seen what perl did internally.

    If you're convinced that something is making an actual difference, run the benchmark a few hundred times (should be quick on a tiny proof of concept program) and do some stats on the results (average and standard deviation especally). This will help show if what you're doing is actually changing anything.

Re: No More Meaningless Benchmarks!
by webfiend (Vicar) on Dec 10, 2006 at 04:08 UTC
    Thank you for sharing this. I don't benchmark my quotes. I benchmark my algorithms. Heck, I don't even worry about benchmarking my algorithms until I know what program I'm supposed to be writing.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://588703]
Approved by Corion
Front-paged by friedo
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (9)
As of 2017-01-19 16:10 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (170 votes). Check out past polls.