Trimming benchmarks

[Tanktalus]

For some unknown reason, I was perusing old posts on StackOverflow, and came across an old benchmark I posted there. Since that time, I've upgraded CPU and installed multiple versions of perl, so I thought I'd see how much of an improvement I got. What I got was a shocker to me. Ok, that's a bit strong - it was still definitely a surprise.

My benchmark was originally tested on a Phenom 2.5GHz system with perl 5.8.8. I then tried on my i7 2.8GHz system with perl 5.12.2. Some of the codepaths improved, but some, like single, went way downhill. So I tried with 5.8.8 on the new box, too, and saw that this was a bigger impact.

I was hoping that regex speed would improve, or at least be statistically similar, from release to release. In some of these cases, however, that does not seem to be the case. I then went and tried all the versions I have and got some widely varying numbers. Is this what others would expect?

perl5.8.8
             Rate    single   capture kramercap     trick    double
single     3596/s        --       -9%       -9%      -95%      -96%
capture    3946/s       10%        --       -0%      -94%      -95%
kramercap  3960/s       10%        0%        --      -94%      -95%
trick     71359/s     1884%     1708%     1702%        --      -11%
double    80563/s     2140%     1942%     1935%       13%        --
perl5.10.1
             Rate    single   capture kramercap     trick    double
single     1857/s        --      -50%      -51%      -98%      -98%
capture    3702/s       99%        --       -1%      -95%      -96%
kramercap  3756/s      102%        1%        --      -95%      -96%
trick     76323/s     4010%     1962%     1932%        --      -10%
double    85132/s     4485%     2200%     2167%       12%        --
perl5.11.5
             Rate    single   capture kramercap     trick    double
single     1820/s        --      -50%      -51%      -98%      -98%
capture    3657/s      101%        --       -1%      -95%      -96%
kramercap  3691/s      103%        1%        --      -95%      -96%
trick     75045/s     4024%     1952%     1933%        --      -11%
double    84148/s     4524%     2201%     2180%       12%        --
perl5.12.0
             Rate    single   capture kramercap     trick    double
single     1734/s        --      -50%      -50%      -98%      -98%
capture    3446/s       99%        --       -1%      -95%      -96%
kramercap  3466/s      100%        1%        --      -95%      -96%
trick     72542/s     4084%     2005%     1993%        --      -10%
double    80991/s     4572%     2250%     2237%       12%        --
perl5.12.1
             Rate    single   capture kramercap     trick    double
single     1749/s        --      -49%      -51%      -98%      -98%
capture    3455/s       98%        --       -2%      -95%      -96%
kramercap  3540/s      102%        2%        --      -95%      -96%
trick     74052/s     4134%     2044%     1992%        --      -10%
double    82732/s     4630%     2295%     2237%       12%        --
perl5.12.2
             Rate    single   capture kramercap     trick    double
single     1972/s        --      -50%      -51%      -97%      -97%
capture    3915/s       99%        --       -2%      -94%      -95%
kramercap  3998/s      103%        2%        --      -94%      -95%
trick     64740/s     3183%     1554%     1519%        --      -11%
double    73100/s     3607%     1767%     1728%       13%        --

Since it's often deemed bad behaviour to link to code elsewhere (though I don't see StackOverflow being quite the same issue), here's a copy of the benchmark code.

#!/usr/bin/perl

use strict;
use warnings;

use Benchmark qw(:all);

my $a = 'a' x 1_000;

my @x = (
         "    $a   ",
         "$a   ",
         $a,
         "    $a"
        );

cmpthese(-5,
         {
             single => sub {
                 for my $s (@x)
                 {
                     my $x = $s;
                     $x =~ s/^\s+|\s+$//g;
                 }
             },
             double => sub {
                 for my $s (@x)
                 {
                     my $x = $s;
                     $x =~ s/^\s+//;
                     $x =~ s/\s+$//;
                 }
             },
             trick => sub {
                 for my $s (@x)
                 {
                     my $x = $s;
                     s/^\s+//, s/\s+$// for $x;
                 }
             },
             capture => sub {
                 for my $s (@x)
                 {
                     my $x = $s;
                     $x =~ s/\A\s*(.*?)\s*\z/$1/
                 }
             },
             kramercap => sub {
                 for my $s (@x)
                 {
                     my $x = $s;
                     ($x) = $x =~ /^\s*(.*?)\s*$/
                 }
             },
         }
        );