Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^4: Loading 283600 records (Updated)

by remiah (Hermit)
on Sep 23, 2012 at 23:10 UTC ( #995269=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Loading 283600 records (Updated)
in thread Loading 283600 records (WordNet)

Added 3 test

05 .. unpack, using List::MoreUtils's natatime
06 .. unpack again
07 .. yours
Your unpack was faster than mines. This is benchmark results.
                   s/iter 02_split1 05_unpack_natatime 06_unpack_map 04_unpack 03_split2 07_unpack_2
02_split1            6.38        --               -13%          -16%      -35%      -42%        -50%
05_unpack_natatime   5.55       15%                 --           -4%      -25%      -33%        -43%
06_unpack_map        5.34       19%                 4%            --      -22%      -31%        -40%
04_unpack            4.18       53%                33%           28%        --      -11%        -24%
03_split2            3.70       72%                50%           44%       13%        --        -14%
07_unpack_2          3.18      100%                74%           68%       31%       16%          --
01_substr            2.70      136%               105%           98%       55%       37%         18%
And test code added.
sub test5{ #with natatime use List::MoreUtils qw/natatime/; $href={}; open(my $fh, "<", "04_A10A10A4.ascii") or die $!; local $/= undef; my $lt = natatime(3, unpack( '(a10a10a4)*', <$fh>)); while( my @rec=$lt->() ){ push @{ $href->{ $rec[0] } }, [ @rec[ 1, 2 ] ] } close $fh; } sub test6{ $href={}; open(my $fh, "<", "04_A10A10A4.ascii") or die $!; local $/= undef; my $i=1; my($s1,$s2); map { $s1=$_ if $i % 3 == 1; $s2=$_ if $i % 3 == 2; if( $i % 3 == 0){ push @{ $href->{$s1} }, [$s2, $_]; } $i++; }unpack( '(a10a10a4)*', <$fh> ); close $fh; } sub test7 { my %hash=(); open(my $fh, "<", "04.txt") or die $!; while( <$fh> ) { my( $k, @v ) = unpack( 'a10a10a4', $_ ); push @{ $hash{ $k } }, \@v } close $fh; }
dsheroh told me of in memory SQLite. It's loading time is apparently faster than any of above tests. I will report it later.


Comment on Re^4: Loading 283600 records (Updated)
Download Code
Replies are listed 'Best First'.
Re^5: Loading 283600 records (substr alias)
by Anonymous Monk on Sep 24, 2012 at 03:57 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://995269]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (14)
As of 2015-07-29 19:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (267 votes), past polls