Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

No Benchmark is better than a bad Benchmark

by Abigail-II (Bishop)
on Aug 19, 2002 at 11:49 UTC ( #191143=note: print w/ replies, xml ) Need Help??


in reply to Re: Re: Optimizing existing Perl code (in practise)
in thread Optimizing existing Perl code (in practise)

You should always be very suspicious if your benchmark shows results of 8823529.41 runs/second. Specially when it comes to non-trivial tasks like sprintf() - after all, than requires perl to parse a format.

Another thing that should ring loud bells is that you are doing sprintf() in void context. That's not a natural operation. Perhaps Perl optimizes that away for you - totally screwing up your benchmark. It's a simple test:

$ perl -MO=Deparse -wce 'sprintf "%x%x%x", ord ("a"), ord ("b"), o +rd ("c")' Useless use of a constant in void context at -e line 1. BEGIN { $^W = 1; } '???'; -e syntax OK $
Indeed, you just benchmarked how fast perl can do an empty loop. Not very useful. Your benchmark should include assigning the result to a variable. So, you might want to do:
#!/usr/bin/perl use strict; use warnings 'all'; use Benchmark; timethese -10 => { unpack => '$_ = unpack "H*" => "abc"', sprintf => '$_ = sprintf "%x%x%x", ord ("a"), ord ("b"), ord ( +"c")', } __END__ Benchmark: running sprintf, unpack for at least 10 CPU seconds... sprintf: 11 wallclock secs (10.25 usr + 0.00 sys = 10.25 CPU) @ 77 +5053.56/s (n=7944299) unpack: 11 wallclock secs (10.48 usr + 0.01 sys = 10.49 CPU) @ 33 +1145.09/s (n=3473712)
It looks like sprintf is still a winner. But is it? Let's check the deparser again:
$ perl -MO=Deparse -wce '$_ = sprintf "%x%x%x", ord "a", ord "b", +ord "c"' BEGIN { $^W = 1; } $_ = '616263'; -e syntax OK $
Oops. Perl is so smart, it figured out at compile time the result of the sprintf. We'd have to make the arguments of sprintf variable to make Perl actually do work at run time:
$ perl -MO=Deparse -wce '($a, $b, $c) = split // => "abc"; $_ = sprintf "%x%x%x", ord $a, ord $b, ord $c' BEGIN { $^W = 1; } ($a, $b, $c) = split(//, 'abc', 4); $_ = sprintf('%x%x%x', ord $a, ord $b, ord $c); -e syntax OK $
And only now we can run a fair benchmark:
#!/usr/bin/perl use strict; use warnings 'all'; use Benchmark; use vars qw /$a $b $c $abc/; $abc = "abc"; ($a, $b, $c) = split // => $abc; timethese -10 => { unpack => '$_ = unpack "H*" => $::abc', sprintf => '$_ = sprintf "%x%x%x", ord $::a, ord $::b, ord $:: +c', } __END__ Benchmark: running sprintf, unpack for at least 10 CPU seconds... sprintf: 11 wallclock secs (10.51 usr + 0.01 sys = 10.52 CPU) @ 20 +8379.75/s (n=2192155) unpack: 10 wallclock secs (10.10 usr + 0.00 sys = 10.10 CPU) @ 32 +3836.04/s (n=3270744)
And guess what? unpack is the winner!

The moral: no benchmark is better than a bad benchmark.

Abigail


Comment on No Benchmark is better than a bad Benchmark
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://191143]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (9)
As of 2014-08-21 17:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (139 votes), past polls