perl 5.36 and the for_list feature - a simple speed comparison

Perl 5.36 has a new feature that iterates over multiple values at a time in a for loop (perldelta entry here). Any number of values can be specified, but the pairwise case is the focus here.

The for_list feature is a useful alternative to a while-each idiom over a hash when working with key-value pairs. It also allows one to natively work with arrays of key-value pairs without converting them to a hash (and thus losing values associated with duplicate keys).

Speed is one factor in the decision to updated code, so I figured I would see how fast the new feature is compared with while-each and some other approaches using List::Util functions. Some benchmark code is below (inside readmore tags), with results following.

Benchmark labels starting with a_ operate on array data while those with h_ operate on hash data.

I note that it is fairly well documented that the while-each idiom is best avoided because the hash iterator has global state and thus can cause action at a distance. In the benchmark code the hash is only accessed this way in one sub.

The summation of the hash values in the loops is merely something for the loop to do that would have relatively low overhead compared with the looping itself.

The hash_vals sub is done as a point of comparison. There is no reason why one would iterate over both keys and values when only the values are needed.

Code was run under Ubuntu 20.04 under WSL2 for Windows.

The main conclusion is that the new feature is faster than all the others when an array is used. It is faster than all of the hash approaches that use both keys and values. Profiling shows the cause of the slowdown when using List::Util::pairs is the dereference of the pair array, which is not surprising. Using the declared_refs feature does not help in this case.

Overall I quite like the new for_list feature. Aside from being faster than many of the alternatives, it is also much cleaner.

use 5.036;
use strict;
use warnings;
use List::Util qw /pairs pairvalues/;
use Benchmark qw /cmpthese/;

use experimental qw/for_list declared_refs/;
    
my @vals = (1..100000);
my %val_hash = @vals;

say lu_pairs();
say lu_pairs_refalias();
say lu_pairvals();
say for_list_a();
say for_list_h();
say while_each();
say hash_vals();
say hash_by_key();

cmpthese(-3, {
    a_pairs       => \&lu_pairs,
    a_pairs_alias => \&lu_pairs_refalias,
    a_pair_vals   => \&lu_pairvals,
    a_for_list    => \&for_list_a,
    h_for_list    => \&for_list_h,
    h_each        => \&while_each,
    h_vals        => \&hash_vals,
    h_by_key      => \&hash_by_key,
});

sub lu_pairs {
    my $i;
    for my $pair (pairs @vals) {
        $i += $pair->[1];
    }
    return $i;
}

sub lu_pairs_refalias {
    my $i;
    for my \@pair (pairs @vals) {
        $i += $pair[1];
    }
    return $i;
}

sub lu_pairvals {
    my $i;
    for my $val (pairvalues @vals) {
        $i += $val;
    }
    return $i;
}


sub for_list_a {
    my $i;
    for my ($key, $value) (@vals) {
        $i += $value;
    }
    return $i;
}

sub for_list_h {
    my $i;
    for my ($key, $value) (%val_hash) {
        $i += $value;
    }
    return $i;
}

sub while_each {
    my $i;
    while (my ($key, $value) = each %val_hash) {
        $i += $value;
    }
    return $i;
}

sub hash_vals {
    my $i;
    foreach my $value (values %val_hash) {
        $i += $value;
    }
    return $i;

}

sub hash_by_key {
    my $i;
    foreach my $key (keys %val_hash) {
        $i += $val_hash{$key};
    }
    return $i;

}
[download]

Results:

2500050000
2500050000
2500050000
2500050000
2500050000
2500050000
2500050000
2500050000
                Rate a_pairs_alias a_pairs h_by_key h_each h_for_list 
+h_vals a_pair_vals a_for_list
a_pairs_alias 87.9/s            --     -9%     -19%   -35%       -51% 
+  -82%        -83%       -89%
a_pairs       96.5/s           10%      --     -11%   -28%       -47% 
+  -80%        -81%       -88%
h_by_key       109/s           24%     13%       --   -19%       -40% 
+  -77%        -78%       -87%
h_each         135/s           53%     40%      24%     --       -25% 
+  -72%        -73%       -84%
h_for_list     181/s          106%     88%      66%    34%         -- 
+  -62%        -64%       -78%
h_vals         477/s          443%    394%     338%   254%       164% 
+    --         -6%       -42%
a_pair_vals    506/s          476%    424%     364%   275%       180% 
+    6%          --       -39%
a_for_list     827/s          841%    757%     659%   513%       357% 
+   73%         63%         --
[download]

Comment on perl 5.36 and the for_list feature - a simple speed comparison Select or Download Code

Replies are listed 'Best First'.
Re: perl 5.36 and the for_list feature - a simple speed comparison by duelafn (Parson) on Jun 21, 2022 at 11:46 UTC
`List::Util::pairmap` would be an existing solution very similar to the new feature. It also seems to run quite quickly (but I can't compare it to for_list since I don't have a new-enough perl). `use List::Util qw/ pairmap /; sub a_pairmap { my $i; pairmap { $i += $b } @vals; return $i; }` [download] Good Day, Dean	[reply] [d/l] [select]
Re^2: perl 5.36 and the for_list feature - a simple speed comparison by kcott (Archbishop) on Jun 21, 2022 at 21:30 UTC
G'day Dean, "`List::Util::pairmap` would be an existing solution very similar to the new feature. It also seems to run quite quickly (but I can't compare it to for_list since I don't have a new-enough perl)." No problem. I have `v5.36.0` installed. I added your code to the OP benchmark code. Mine starts as follows. Refer to "perl5360delta: use v5.36" for why I removed the `strict` and `warnings` pragmata. `#!/usr/bin/env perl use v5.36; use List::Util qw /pairs pairvalues pairmap/;` [download] I then added: `say a_pairmap();` after the existing eight `say` statements. `a_pairmap => \&a_pairmap,` to the end of the `cmpthese` hashref. Your `sub a_pairmap { ... }` verbatim, at the end of the script. I ran the benchmark five times. All results were pretty much the same; here's a median set: 2500050000 2500050000 2500050000 2500050000 2500050000 2500050000 2500050000 2500050000 2500050000 Rate a_pairs_alias a_pairs h_each h_by_key h_for_list +h_vals a_pair_vals a_pairmap a_for_list a_pairs_alias 48.8/s -- -4% -67% -73% -83% + -90% -90% -92% -94% a_pairs 51.0/s 4% -- -66% -71% -82% + -90% -90% -91% -94% h_each 150/s 207% 193% -- -16% -47% + -69% -71% -74% -82% h_by_key 178/s 265% 249% 19% -- -37% + -64% -65% -69% -78% h_for_list 284/s 482% 457% 90% 59% -- + -42% -45% -51% -66% h_vals 490/s 903% 860% 227% 175% 72% + -- -5% -16% -41% a_pair_vals 513/s 951% 906% 243% 188% 81% + 5% -- -12% -38% a_pairmap 582/s 1091% 1041% 289% 226% 105% + 19% 13% -- -30% a_for_list 827/s 1594% 1521% 453% 364% 191% + 69% 61% 42% -- [download] In all runs, `a_pairmap` was second fastest; and, `h_by_key` was faster than `h_each` (which differs from the OP results). My environment: `$ perl -v \| head -2 \| tail -1 This is perl 5, version 36, subversion 0 (v5.36.0) built for cygwin-th +read-multi $ uname -a CYGWIN_NT-10.0 titan 3.2.0(0.340/5/3) 2021-03-29 08:42 x86_64 Cygwin` [download] — Ken	[reply] [d/l] [select]
Re^3: perl 5.36 and the for_list feature - a simple speed comparison by swl (Vicar) on Jun 21, 2022 at 22:51 UTC
Thanks for this. Updated code and results are in 11144924.	[reply]
Re^2: perl 5.36 and the for_list feature - a simple speed comparison by swl (Vicar) on Jun 21, 2022 at 22:49 UTC
Thanks for that. List::Util::pairmap is a good suggestion. Below is some updated code to add your array code and a hash variant. The results of three runs follow. The conclusions wrt pairmap are pretty much the same as those of kcott in 11144922. A point of difference is that `h_by_key` remains slower in my case. I wonder if there are differences due to the compiler used to build the perls or there's something cygwin vs linux. My system details are at the end. Edit - the original code used `say a_pairmap()` twice. kcott++ for noting. Read more... (2 kB) Rate a_pairs_alias a_pairs h_by_key h_pairmap h_each h +_pair_vals h_for_list h_vals a_pair_vals a_pairmap a_for_list a_pairs_alias 83.5/s -- -7% -13% -33% -34% + -41% -53% -79% -83% -84% -90% a_pairs 89.5/s 7% -- -6% -28% -29% + -37% -50% -78% -82% -83% -89% h_by_key 95.5/s 14% 7% -- -23% -25% + -33% -46% -76% -81% -81% -88% h_pairmap 125/s 49% 39% 30% -- -2% + -12% -30% -69% -75% -76% -85% h_each 127/s 52% 41% 33% 2% -- + -11% -29% -68% -74% -75% -84% h_pair_vals 142/s 70% 59% 49% 14% 12% + -- -20% -65% -71% -72% -82% h_for_list 178/s 113% 99% 86% 43% 41% + 25% -- -56% -64% -65% -78% h_vals 401/s 380% 348% 320% 222% 217% + 182% 125% -- -18% -22% -50% a_pair_vals 490/s 487% 448% 413% 294% 287% + 245% 175% 22% -- -4% -39% a_pairmap 512/s 513% 472% 436% 311% 304% + 260% 188% 28% 4% -- -36% a_for_list 804/s 863% 799% 742% 546% 535% + 466% 352% 101% 64% 57% -- Rate a_pairs_alias a_pairs h_by_key h_each h_pair_vals + h_pairmap h_for_list h_vals a_pair_vals a_pairmap a_for_list a_pairs_alias 73.2/s -- -17% -25% -31% -49% + -49% -54% -82% -85% -86% -91% a_pairs 87.7/s 20% -- -10% -17% -39% + -39% -45% -79% -82% -83% -89% h_by_key 97.0/s 33% 11% -- -8% -32% + -32% -39% -77% -81% -82% -88% h_each 106/s 44% 21% 9% -- -26% + -26% -34% -75% -79% -80% -87% h_pair_vals 143/s 96% 63% 47% 35% -- + -0% -11% -66% -71% -73% -82% h_pairmap 143/s 96% 63% 48% 36% 0% + -- -11% -65% -71% -73% -82% h_for_list 160/s 119% 83% 65% 52% 12% + 12% -- -61% -68% -70% -80% h_vals 415/s 467% 373% 328% 292% 190% + 189% 159% -- -17% -22% -48% a_pair_vals 498/s 581% 468% 413% 371% 248% + 248% 211% 20% -- -6% -38% a_pairmap 531/s 626% 506% 448% 403% 271% + 271% 231% 28% 7% -- -33% a_for_list 797/s 990% 809% 722% 654% 457% + 456% 398% 92% 60% 50% -- Rate a_pairs_alias a_pairs h_by_key h_each h_pairmap h +_pair_vals h_for_list h_vals a_pair_vals a_pairmap a_for_list a_pairs_alias 75.9/s -- -18% -29% -41% -41% + -52% -58% -77% -83% -86% -89% a_pairs 92.4/s 22% -- -13% -28% -29% + -41% -49% -72% -80% -82% -87% h_by_key 107/s 40% 16% -- -17% -18% + -32% -41% -67% -77% -80% -85% h_each 128/s 69% 39% 20% -- -1% + -18% -29% -60% -72% -76% -82% h_pairmap 130/s 71% 40% 21% 1% -- + -17% -28% -60% -72% -75% -82% h_pair_vals 157/s 106% 70% 47% 22% 21% + -- -13% -52% -66% -70% -78% h_for_list 180/s 137% 95% 69% 40% 39% + 15% -- -45% -61% -66% -74% h_vals 325/s 328% 252% 204% 153% 151% + 107% 80% -- -29% -38% -54% a_pair_vals 459/s 504% 397% 330% 257% 254% + 193% 155% 41% -- -13% -35% a_pairmap 527/s 594% 470% 394% 311% 306% + 236% 192% 62% 15% -- -25% a_for_list 705/s 828% 663% 560% 449% 444% + 350% 291% 117% 54% 34% -- [download] `perl -v \| head -2 \| tail -1 This is perl 5, version 36, subversion 0 (v5.36.0) built for x86_64-li +nux uname -a Linux L-933C0F3 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00 +:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux gcc -v 2>&1 \| tail -2 Thread model: posix gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)` [download]	[reply] [d/l] [select]


We don't bite newbies here... much
	PerlMonks