Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: Fastest data structure compare?

by eserte (Deacon)
on Oct 08, 2009 at 10:58 UTC ( #799931=note: print w/ replies, xml ) Need Help??


in reply to Fastest data structure compare?

Using Storable is problematic, because it's not guaranteed that semantically equivalent data is serialized in the same way. Here's an example playing with the internal utf-8 flag --- while Test::More's is_deeply and Data::Compare get it right, checking the serialized Storable data fails:

#!/usr/bin/perl use Data::Compare qw(); use Storable qw(nfreeze); use Test::More qw(no_plan); my $data1 = ["f\xfcbar"]; my $data2 = [substr("f\xfcbar\x{0100}", 0, -1)]; is_deeply($data1, $data2, "is_deeply test"); ok(Data::Compare::Compare($data1, $data2), "Data::Compare"); ok(nfreeze($data1) eq nfreeze($data2), "storable serialized");
I just benchmarked Test::More::is_deeply vs. Data::Compare and found that the latter is 3x faster for a data set which size is ~6MB as a storable-serialized file. This probably depends on the structure of the data set.


Comment on Re: Fastest data structure compare?
Download Code
Re^2: Fastest data structure compare?
by Anonymous Monk on Oct 08, 2009 at 12:30 UTC
    Storable warns against this
    if you happen to use your numbers as strings between two freezing operations on the same data structures, you will get different results.

    There is no facility either to return all strings as utf8 sequences, or to attempt to convert utf8 data back to 8 bit and croak() if the conversion fails.

    It is easily avoided if you use encoding/Encode
      Encode may help in the utf8 case, but not in the case of different integer representation (as mentioned in the Storable manpage):
      #!/usr/bin/perl use Data::Compare qw(); use Storable qw(nfreeze); use Test::More qw(no_plan); my $data1 = { foobar => 1 }; my $data2 = { foobar => "1" }; $Storable::canonical = 1; is_deeply($data1, $data2, "is_deeply test"); ok(Data::Compare::Compare($data1, $data2), "Data::Compare"); ok(nfreeze($data1) eq nfreeze($data2), "storable serialized");
        Great, you've read the bug section :)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://799931]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (14)
As of 2014-12-19 20:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (91 votes), past polls