http://www.perlmonks.org?node_id=942747

spx2 has asked for the wisdom of the Perl Monks concerning the following question:

Hello monks,

Statement of the problem

Basically I'm in a situation where I need to find a fast way to serialize/deserialize many arrays of 6 elements each. These arrays have the same semantics inside, I'm just supposed to transform it into some form so I can pass it to other methods/modules etc.

The problem is I need to do this faster than the current implementation(splits a string which contains numbers, puts those numbers into the values of a hash).

At the moment these arrays of 6 elements are from some data source where they come as comma-separated strings, so a split(/,/,$string) can't be avoided.

I've identified that these are the data types that I'll be getting.

Basically the data which needs to be serialized is consisting of 2integers,an unsigned long,3 double precision floats, in this order.

So I get a string like that, I decompose it into bits and pieces, serialize it, add it to an array and after I processed all the strings, I return that big array. Of course, on the other end, I'm getting the array and now I need to deserialize in order to access the values.

Now the question is, which is the fastest way to serialize/deserialize data. So what I did is I made a benchmark to see exactly how fast each is. I was interested in how fast the creation of the object was(serialization), and how fast access to the object was(which in some cases required deserialization).

Questions

I'm gonna add the code which I wrote to benchmark this, but this has left me with some unanswered questions:

  1. Why is packing and returning six values slower than constructing a whole Perl AV data structure, creating each SvPV for each position, copying all the elements inside it and then returning that ? (compare the times of construct_packed with construct_array)
  2. Why is unpacking a blob of 6 values to 6 different SV structures(among which 2 SvIV and 3 SvPVNV and one SvUV ) slower than dereferencing a whole array and copying contents to 6 different scalars(the same types mentioned before) ? (compare times of access_array and access_packed)
  3. Why is accessing a hash faster than just unpacking a string filled with data and accessing the contents ? (compare access_packed and access_hash)

I am asking this because when I first started doing the benchmark I was thinkg "Oh boy! I'm gonna optimize this thing by using (un)pack for (de)serialization instead of hashes or arrays" so that's what I was expecting and what I got is waaay off my initial expectation and I don't know where I'm wrong.

Possible improvements(haven't tried them)

I also thought of an alternative to all of this, that's to change the code that generates the data from the data source so that it gives data of fixed size in binary format instead of text.

I also thought about ditching unpack for a homebrewed much more limited XS version of it(haven't worked on it yet). Would it, in principle, be possible to write something than the current unpack ?

Benchmark result

user@garage:~$ perl pack_vs_hash_pm.pl
                               Rate construct_comma_separated construct_hash construct_packed construct_array
construct_comma_separated  699301/s                        --           -13%             -43%            -70%
construct_hash             800000/s                       14%             --             -35%            -66%
construct_packed          1234568/s                       77%            54%               --            -47%
construct_array           2325581/s                      233%           191%              88%              --
            (warning: too few iterations for a reliable count)
                            Rate access_comma_separated access_packed access_hash access_array
access_comma_separated  970874/s                     --          -23%        -33%         -69%
access_packed          1265823/s                    30%            --        -13%         -59%
access_hash            1449275/s                    49%           14%          --         -54%
access_array           3125000/s                   222%          147%        116%           --


Code

#!/usr/bin/perl use strict; use warnings; use Benchmark qw/cmpthese/; ################## # SERIALIZING # ################## cmpthese( 1_000_000, { construct_hash => sub { my @a = (1..6); return +{ k1 => $a[0], k2 => $a[1], k3 => $a[2], k4 => $a[3], k5 => $a[4], k6 => $a[5] } }, construct_packed => sub { my @a = (1..6); return pack('sI2Lddd', 6, @a );# ignore the front 's' , I'm usi +ng that for something else }, # old cache construct_comma_separated => sub { my @a = (1..6); return "$a[0],$a[1],$a[2],$a[3],$a[4],$a[5]"; }, construct_array => sub { my @a = (1..6); return \@a; }, }); ################## # DESERIALIZING # ################## my @access_vals = 1..6; my $access_vals_ref = \@access_vals; my $access_h = +{ k1 => $access_vals[0], k2 => $access_vals[1], k3 => $access_vals[2], k4 => $access_vals[3], k5 => $access_vals[4], k6 => $access_vals[5] }; my $access_packed = pack("sI2Lddd",@access_vals); # ignore the front ' +s' , I'm using that for something else my $access_comma_separated = "3411,2,3,45,5.33,6.12"; cmpthese( 1_000_000 , { access_hash => sub { my $k1 = $access_h->{k1}; my $k2 = $access_h->{k2}; my $k3 = $access_h->{k3}; my $k4 = $access_h->{k4}; my $k5 = $access_h->{k5}; my $k6 = $access_h->{k6}; }, access_packed => sub { my ($values_packed,$k1,$k2,$k3,$k4,$k5,$k6)=unpack('sI2Lddd',$ac +cess_packed);# ignore the front 's' , I'm using that for something el +se }, access_comma_separated => sub { my ($k1,$k2,$k3,$k4,$k5,$k6) = split(/,/,$access_comma_separated +); }, access_array => sub { my ($k1,$k2,$k3,$k4,$k5,$k6) = @$access_vals_ref; }, });