Unexplained benchmark when using chop vs. chomp (or neither)

[Cristoforo]

Solved: See post Re: Unexplained benchmark when using chop vs. chomp (or neither). Thanks to everyone who helped trying to solve this question. And davido avoided the problem by making a copy of the array to operate upon.

I don't understand the different results I am getting when doing a benchmark on 4 different code snippets.

When using chop in the code, the benchmark runs extremely fast. But, when using chomp or neither chop or chomp, the the particular code runs the slowest in the benchmark.

Any ideas?

#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw/first reduce/;
use Benchmark qw/ cmpthese /;

my $file = 'bench_reduce.txt';
open my $fh, '<', $file or die $!;
my @data = <$fh>;
seek $fh, 0, 0;

my @chunks;
{
    local $/ = "";
    @chunks = <$fh>;
}

my $cache = sub {
    my %c; # cache
    my $max = reduce {($c{$a}//=f_32($a)) > ($c{$b}//=f_32($b)) ? $a :
+ $b} @chunks;
    ($max) = $max =~ /\A(.+)/;
    #print $max;
};

my $reduce = sub {
    my $max = reduce {f_32($a) > f_32($b) ? $a : $b} @chunks;
    ($max) = $max =~ /\A(.+)/;
    #print $max;
};


sub f_32 {
    $_[0] =~ /^32 (\S+)/m;
    return $1;    
}

my $tshiono = sub {
    my ($maxline, $firstline);
    my $max = 0;
    my $target = 32;
    for (@data) {
        next unless /\S/;
        chop; # also tested with chomp and with neither chop nor chomp
        if (/^\d+:/) {                  # treat the 1st line
            $firstline = $_;            # keep the 1st line
        } elsif (/^(\d+) +([\d.]+)/) {  # treat other lines in the blo
+ck
            if ($1 == $target && $2 > $max) {
                $max = $2;              # update the highest value
                $maxline = $firstline;  # and the 1st line of the bloc
+k
            }
        }
    }
};

my $_7stud = sub {
    my %results;

    for my $block (@chunks) {
        my @lines = split /\n/, $block;
        my $line32 = first {/^32/} @lines;

        (undef, my $ranking_num) = split ' ', $line32;
        $results{$ranking_num} = $lines[0];
    }

    my @sorted_nums = sort{$a <=> $b} keys %results;

    my $biggest_num = $sorted_nums[-1];
    
};

cmpthese  (-1,
    {
        w_o_cache => $reduce,
        w_cache => $cache,
        tshiono => $tshiono,
        '7stud' => $_7stud
    });

__END__
C:\Old_Data\perlp>perl test2.pl (with chop tshiono)
            Rate     7stud w_o_cache   w_cache   tshiono
7stud      200/s        --      -37%      -45%      -92%
w_o_cache  320/s       60%        --      -12%      -87%
w_cache    364/s       82%       14%        --      -86%
tshiono   2557/s     1180%      700%      602%        --

C:\Old_Data\perlp>perl test2.pl (with chomp tshiono)
           Rate   tshiono     7stud w_o_cache   w_cache
tshiono   160/s        --      -19%      -50%      -57%
7stud     197/s       23%        --      -38%      -47%
w_o_cache 318/s       98%       61%        --      -15%
w_cache   373/s      133%       90%       17%        --

C:\Old_Data\perlp>perl test2.pl (w/o chop or chomp tshiono)
           Rate   tshiono     7stud w_o_cache   w_cache
tshiono   177/s        --      -13%      -45%      -52%
7stud     202/s       15%        --      -37%      -45%
w_o_cache 321/s       82%       59%        --      -13%
w_cache   370/s      109%       83%       15%        --

The file I ran it against was (1000 chunks - 5000 lines):

1: 1.0892 4.4110
33 17.46216
32 32.64771
34 112.64038

2: 9.6591 9.9243
33 116.30859
32 38.09204
34 117.57813

3: 1.6385 3.0847
33 45.08057
32 3.52173
34 145.28809

4: 7.0053 9.0564
33 24.13330
32 169.46411
34 92.74902

5: 9.9045 4.8688
33 130.81055
32 4.88892
34 54.73022

6: 0.1675 1.0004
33 11.64551
32 124.31641
34 186.35864

7: 5.5832 8.3316
33 113.65356
32 23.00415
34 85.02808

8: 5.2414 9.9789
33 37.54883
32 102.96021
34 30.73730

9: 9.4675 2.2897
33 105.28564
32 63.26904
34 11.12671

10: 5.4352 3.6444
33 68.45703
32 140.71655
34 77.05078

11: 5.9378 0.5676
33 70.95947
32 8.36182
34 162.20703

12: 7.9144 1.4954
33 37.33521
32 42.61475
34 160.79102

13: 2.7045 9.0518
33 28.79028
32 152.38647
34 23.10791

14: 3.0066 9.0158
33 89.53857
32 161.81030
34 17.35840

15: 5.8575 3.3990
33 16.30249
32 14.50806
34 53.58887

16: 2.2849 0.5002
33 189.22729
32 52.99072
34 103.76587

17: 9.3036 2.6373
33 159.46655
32 76.39160
34 42.39502

18: 0.4990 8.1360
33 81.29883
32 65.38086
34 183.02612

19: 2.6181 9.0866
33 68.85986
32 19.53735
34 117.96875

20: 5.5716 7.2162
33 67.04102
32 151.03760
34 93.67065

21: 6.4343 9.7278
33 105.16357
32 85.71167
34 195.94727

22: 2.1356 3.5342
33 79.92554
32 78.86353
34 183.61206

23: 6.0007 9.6381
33 127.00806
32 171.78955
34 79.93164

24: 8.6444 5.3683
33 167.31567
32 187.07275
34 59.71680

25: 7.6883 5.6271
33 82.87964
32 23.54736
34 171.41724

26: 7.1753 5.2127
33 118.62183
32 102.74658
34 17.41333

27: 6.3071 9.0173
33 160.80322
32 37.37183
34 112.35962

28: 9.8917 1.8271
33 55.63965
32 30.85327
34 193.15186

29: 9.3063 1.8320
33 89.63623
32 138.42163
34 4.58984

30: 8.0484 7.4863
33 142.09595
32 192.08984
34 151.72119

31: 3.5632 1.7795
33 118.94531
32 4.26025
34 49.07227

32: 9.5773 4.7897
33 0.84839
32 93.93921
34 105.06592

33: 7.7533 9.5587
33 115.83252
32 168.96362
34 50.00610

34: 1.0129 8.2678
33 174.52393
32 77.58789
34 84.80225

35: 4.4803 8.4412
33 31.43921
32 99.33472
34 158.77686

36: 9.9988 4.5566
33 53.35693
32 119.68384
34 181.53076

37: 7.7020 2.0383
33 32.53174
32 7.19604
34 16.04614

38: 7.1399 7.3541
33 147.31445
32 107.27539
34 45.70923

39: 8.7286 2.9053
33 189.21509
32 174.50562
34 11.57227

40: 1.7572 5.9695
33 23.92578
32 72.00928
34 42.18750

41: 4.4919 3.3966
33 83.71582
32 146.63696
34 45.71533

42: 6.4484 5.2805
33 92.90771
32 90.19775
34 188.25073

43: 1.0941 2.2470
33 181.93970
32 118.30444
34 69.34814

44: 1.4545 6.5723
33 119.01855
32 54.29077
34 108.66699

45: 0.1193 6.9247
33 14.95972
32 21.35010
34 124.39575

46: 2.9922 9.3030
33 110.78491
32 23.29712
34 135.34546

47: 1.0715 4.1751
33 75.29297
32 49.02344
34 131.18896

48: 1.1270 9.5557
33 89.06860
32 89.55688
34 188.93433

49: 6.8298 0.5603
33 169.64722
32 42.34009
34 195.75806

50: 3.7204 8.0731
33 12.04834
32 182.76367
34 30.04761

51: 6.4676 2.4835
33 108.92334
32 145.84961
34 65.14282

52: 4.7330 5.5518
33 45.87402
32 93.24951
34 84.01489

53: 7.3666 2.3865
33 183.98438
32 37.37793
34 67.19971

54: 8.9166 1.5652
33 48.73047
32 161.04126
34 149.40796

55: 2.5650 5.8023
33 196.39893
32 74.03564
34 157.56226

56: 1.9995 2.3358
33 5.43213
32 69.85474
34 123.12622

57: 5.9445 3.9627
33 12.15820
32 100.65918
34 85.90698

58: 7.0340 7.8214
33 34.12476
32 130.82886
34 37.51831

59: 3.4177 9.9609
33 8.33130
32 60.53467
34 111.27930

60: 4.9487 6.5036
33 112.33521
32 64.58130
34 33.66089

61: 9.0079 4.3008
33 112.75635
32 158.45947
34 66.91895

62: 1.7398 3.0731
33 60.40039
32 117.28516
34 13.95264

63: 6.8384 9.2804
33 119.94019
32 176.25122
34 23.84033

64: 7.4100 6.1813
33 2.13623
32 101.34888
34 52.11182

65: 9.8358 2.2983
33 127.81982
32 57.03125
34 54.39453

66: 3.9740 6.8088
33 119.72046
32 159.64966
34 13.79395

67: 9.6390 7.1008
33 4.36401
32 81.16455
34 177.09961

68: 2.8333 8.8361
33 57.78809
32 38.42163
34 82.33643

69: 6.0590 0.4404
33 85.68115
32 79.03442
34 197.29614

70: 6.8454 8.3401
33 35.11353
32 143.61572
34 15.61279

71: 4.6860 0.8606
33 152.88696
32 188.64746
34 169.35425

72: 0.7166 3.2431
33 16.52222
32 172.79053
34 7.27539

73: 4.1467 7.5842
33 58.91724
32 94.75098
34 120.82520

74: 6.1691 6.4380
33 7.37915
32 185.61401
34 13.57422

75: 6.3385 2.0398
33 103.37524
32 105.09644
34 21.33179

76: 6.9598 9.6793
33 83.34961
32 124.68262
34 160.47363

77: 9.9326 3.6383
33 108.58154
32 54.29077
34 74.77417

78: 4.4620 8.5956
33 184.26514
32 150.34180
34 189.88037

79: 5.8237 6.2778
33 58.19092
32 101.26953
34 187.26196

80: 8.4534 7.3975
33 7.44629
32 33.40454
34 63.59253

81: 9.2337 3.9478
33 46.09375
32 27.05078
34 40.09399

82: 1.2070 1.4856
33 158.13599
32 182.67822
34 43.94531

83: 3.1189 3.9801
33 72.77832
32 71.49048
34 102.23999

84: 7.2223 3.4537
33 77.67334
32 195.70313
34 36.21216

85: 7.6334 6.2009
33 43.85376
32 94.72656
34 74.91455

86: 7.2015 9.1669
33 198.40088
32 35.39429
34 134.37500

87: 6.8979 6.7264
33 96.20361
32 136.99341
34 197.21680

88: 4.4299 1.9403
33 93.53638
32 178.04565
34 184.14307

89: 3.4857 7.6654
33 26.01929
32 104.07104
34 81.26831

90: 5.7971 0.6119
33 125.03052
32 73.73047
34 190.13672

91: 9.1089 9.7906
33 44.94019
32 149.63989
34 12.77466

92: 9.5602 7.6926
33 67.71240
32 133.89893
34 172.46094

93: 3.2031 5.5722
33 70.59326
32 185.66284
34 73.77319

94: 8.8495 3.1110
33 29.21143
32 138.22632
34 47.12524

95: 5.7419 6.0880
33 191.05835
32 52.31323
34 143.01758

96: 5.0851 3.5330
33 116.52222
32 157.37305
34 89.64233

97: 2.8961 8.4531
33 154.04663
32 44.94019
34 131.50635

98: 0.1855 3.1180
33 147.65625
32 147.00928
34 166.31470

99: 0.3491 1.8594
33 46.12427
32 152.19116
34 40.74097

100: 1.9360 5.1099
33 94.50684
32 4.43726
34 8.89282

...

980: 7.6590 0.1218
33 186.43188
32 180.46265
34 36.37085

981: 1.3202 3.3847
33 155.68237
32 18.16406
34 65.32593

982: 3.9047 5.1355
33 163.06152
32 110.54077
34 189.04419

983: 8.5547 6.5576
33 155.72510
32 197.72339
34 134.64966

984: 0.2481 2.0639
33 18.96362
32 40.72876
34 166.73584

985: 7.3749 1.5366
33 98.92578
32 57.36694
34 42.84668

986: 7.7145 8.2721
33 148.16895
32 93.54858
34 50.45166

987: 3.9731 7.8418
33 140.75928
32 113.00659
34 7.70874

988: 1.9385 2.5787
33 40.78369
32 53.44849
34 185.05249

989: 2.5055 3.4680
33 86.19995
32 158.31909
34 34.89380

990: 2.1933 2.4203
33 197.74780
32 188.86108
34 14.01978

991: 4.7644 1.2238
33 88.16528
32 132.70264
34 24.19434

992: 8.9307 0.8044
33 139.25781
32 149.59106
34 19.18945

993: 4.4110 0.4428
33 80.05981
32 18.20068
34 95.30640

994: 2.4542 7.6740
33 80.09033
32 69.81812
34 25.07324

995: 9.3329 5.8911
33 82.72095
32 194.39697
34 95.65430

996: 0.8450 3.8217
33 32.95288
32 43.08472
34 18.46924

997: 2.0261 6.6580
33 68.61572
32 52.85645
34 136.04736

998: 2.7808 2.9456
33 54.71191
32 75.15869
34 15.55786

999: 0.6631 9.5544
33 161.90186
32 185.33325
34 61.21216

1000: 1.6800 5.0317
33 94.12842
32 188.36060
34 6.04248

[davido]

The following code strives to test only chop, chomp, and a noop for the same number of iterations, with the same starting data on each iteration. As far as is practical it attempts to maintain simplicity and to establish a control group (mynada) that performs as similar an algorithm as possible to the two other alternatives. Observe this code and the results:

 use strict;                          
 use warnings;                        
                                      
 use Benchmark qw(cmpthese);          
                                      
                                      
 my @orig = ("\n") x 100000;          
                                      
                                      
 sub mychomp {                        
     my $data = shift;                
     my @copy = @$data;               
     chomp for @copy;                 
 }                                    
                                      
 sub mychop {                         
     my $data = shift;                
     my @copy = @$data;               
     chop for @copy;                  
 }                                    
                                      
 sub mynada {                         
     my $data = shift;                
     my @copy = @$data;               
     1 for @copy;                     
 }                                    
                                      
                                      
 cmpthese -10, {                      
     chomp   => sub {mychomp(\@orig)},
     chop    => sub {mychop(\@orig)}, 
     nada    => sub {mynada(\@orig)}, 
 };

This produces (on my system):

        Rate chomp  chop  nada
chomp 87.8/s    --   -4%  -20%
chop  91.6/s    4%    --  -16%
nada   109/s   25%   20%    --

This shows me that the noop is about 20%-25% faster than the other two alternatives, and that the other two are only 4% apart, which is really within the margin for error.

The fact that you are getting more dramatically different results might indicate that the experiment is not as pristine as it should be, and that you are comparing non-equivalent alternatives.

[choroba]

Note that both chop and chomp can take an array as the argument. Interestingly,

chop @copy;

makes mychomp and mychop just as fast as nada.

[davido]

Yes, I had considered setting up alternatives that pass an array to chomp and chop, but figured it wasn't going to be a very useful comparison for this specific situation. But absolutely, pushing the loop into internals is going to be a lot faster. In fact it wouldn't have surprised me if you had said that the array version was even faster than the 'nada' version, since the 'nada' version still has a loop, unless it's been optimized away, which I'm not sure on.

[ikegami]

A better way of reading your results:

Each chomp took 1/(87.8/s)/100000 = 114 ns (including overhead)
Each chop took 1/(91.6/s)/100000 = 109 ns (including overhead)
Each nada took 1/(109/s)/100000 = 92 ns

so

Each chomp took 22 ns
Each chop took 17 ns

It's not surprising that chop is faster than chomp (since what it does is far, far simpler), but they are both seriously fast! What this means is that trying to optimize this is a waste of time; there is far more "fat" to trim elsewhere.

[Eily]

chop will remove a character from the string each time, unless there is none. Since for aliases the variable to the list elements, the strings in @data get chopped until they are empty. As a matter of fact, since the lines are pretty short, the strings are empty before the 20th call to the function, and even without chopping, the function gets called more than 150 times

Edit: reformulated and shortened my confusing post :)

[Cristoforo]

Thank you Eily. You have shown the problem that caused the results of the benchmark to be what they were.