Interpreting Benchmark's "cmpthese" and "timethese"

[neversaint]

Dear Masters,
I have the following benchmark snippets.

my $r = timethese(
    -5,
    {   list => sub { code1($fstr); },
        enum => sub { code2($fstr); }
    }
);

cmpthese $r;

Now I have difficulty in interpreting the result below:

Benchmark: running enum, list for at least 5 CPU seconds...
      enum:  5 wallclock secs ( 3.55 usr +  1.74 sys =  5.29 CPU) @ 50
+1.89/s (n=2655)
      list:  5 wallclock secs ( 5.32 usr +  0.00 sys =  5.32 CPU) @ 16
+68.23/s (n=8875)

       Rate enum list
enum  502/s   -- -70%
list 1668/s 232%   --

My questions are:

1. from 'cmpthese' results it is clear that "list" is more than twice faster than "enum". But looking at the wallclock value from 'timethese' it shows that both of them are close to each other in their total time, even more the usertime for "enum" is shorter than "list". Now which is the correct benchmark should we use 'cmpthese' or 'timethese'? And which one is actually faster "enum" or "list"?
   
2. What does "n" value refer to in 'timethese' output?

[rhesa]

1. You passed -5 to timethese, which means it should run the benchmark not for a specific number of iterations, but for a specific number of seconds. This is why you see that the wallclock for both "enum" and "list" is exactly 5 seconds. I can't tell why "enum" uses more sys time without seeing the code, but the important bit is that both run for at least 5 cpu seconds
2. The n value shows you the number of iterations timethese managed to run within the specified time. That means that "list" did three times as many iterations as "enum"
3. Since "list" can do 1668 iterations per second, while "enum" only does 502 iterations per second, "list" is over three times as fast.
4. All cmpthese does is take the results from timethese and generate the comparison table at the bottom

[neversaint]

Dear rhesa,


You passed -5 to timethese, which means it should run the benchmark not for a specific number of iterations, but for a specific number of seconds. This is why you see that the wallclock for both "enum" and "list" is exactly 5 seconds. I can't tell why "enum" uses more sys time without seeing the code, but the important bit is that both run for at least 5 cpu seconds

So what is the best way to show the speed comparison in terms of it's time and not iteration/time?
I tried "timediff" as explained in the first part of Benchmark documentation. But it gives the same results as "timethese".

[tye]

So what is the best way to show the speed comparison in terms of it's time and not iteration/time?

Since you have

       Rate enum list
enum  502/s   -- -70%
list 1668/s 232%   --

The times are 1/502 of a second vs. 1/1668 of a second. Or about 0.002s vs. 0.0006s so a difference of 0.0014s.

The block of output that I quoted compares the results while the prior output logs some details about how the computations were done. Those details aren't particularly useful for making a comparison.

You could also multiply by 1000 and say that, if you needed to call the compared bits of code 1000 times each, then one would take about 2s plus overhead while the other would take about 0.6s plus overhead. So the time difference for 1000 iterations would be about 1.4s. Note that the time for "overhead" will usually be quite a bit more than the time required to run just the compared code so that real code doing something 1000 times might have run times like 22s vs 20.6s.

Update:

from 'cmpthese' results it is clear that "list" is more than twice faster than "enum"

Actually, the "232%" means that one is more than three times faster than the other. This is a pet peave of one of my coworkers. There are several contradicting ways to report "percentage difference" and most of them lead to people making mistakes when they try to combine them. He strongly believes that it is best to avoid "percentage difference" and only ever deal in terms of simple multipliers. So he'd report the above results like:

       Rate enum list
enum  502/s   --  0.3
list 1668/s 3.32   --

"50% faster" means "can do it as fast as the other plus 50%" or "150% as fast" or "1.5 times as fast". So "200% faster" means "3 times as fast".

To see how confusing this can be, consider these two ways of describing the same situation:

If "A is 50% faster than B and B is 50% faster than C", how much faster is A than C? 'Obviously', "A is 125% faster than C".

If "A is 1.5 times faster than B and B is 1.5 times faster than C", how much faster is A than C? Well, 1.5*1.5 is 2.25 so, obviously, "A is 2.25 times faster than C".

[rhesa]

So what is the best way to show the speed comparison in terms of it's time and not iteration/time?

I'm not sure what you mean by this. Would timethese with a positive value mean more to you? You could run the benchmark for, say, 2000 iterations instead of for 5 seconds. You'd probably see that "list" will run for about 7 seconds, while "enum" will take about 20 seconds.