Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^3: Loading 283600 records (Updated)

by BrowserUk (Pope)
on Sep 23, 2012 at 16:05 UTC ( #995214=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Loading 283600 records (Updated)
in thread Loading 283600 records (WordNet)

There are no rules -- beyond minimising the number of opcodes called -- that apply in all situations. Try plugging this into your benchmark:

my %hash; while( <> ) { my( $k, @v ) = unpack( 'a10a10a4', $_ ); push @{ $hash{ $k } }, \@v }

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong


Comment on Re^3: Loading 283600 records (Updated)
Download Code
Re^4: Loading 283600 records (Updated)
by remiah (Hermit) on Sep 23, 2012 at 23:10 UTC

    Added 3 test

    05 .. unpack, using List::MoreUtils's natatime
    06 .. unpack again
    07 .. yours
    
    Your unpack was faster than mines. This is benchmark results.
                       s/iter 02_split1 05_unpack_natatime 06_unpack_map 04_unpack 03_split2 07_unpack_2
    02_split1            6.38        --               -13%          -16%      -35%      -42%        -50%
    05_unpack_natatime   5.55       15%                 --           -4%      -25%      -33%        -43%
    06_unpack_map        5.34       19%                 4%            --      -22%      -31%        -40%
    04_unpack            4.18       53%                33%           28%        --      -11%        -24%
    03_split2            3.70       72%                50%           44%       13%        --        -14%
    07_unpack_2          3.18      100%                74%           68%       31%       16%          --
    01_substr            2.70      136%               105%           98%       55%       37%         18%
    
    And test code added. dsheroh told me of in memory SQLite. It's loading time is apparently faster than any of above tests. I will report it later.

Re^4: Loading 283600 records (Updated)
by remiah (Hermit) on Sep 24, 2012 at 10:31 UTC

    Hello, BrowserUK.

    I posted some benchmarks of sqlite in-memory at the bottom of this thread. It was surprise for me. Please have a look at.

    and thanks for responding to me.
    regards.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://995214]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (8)
As of 2014-12-20 18:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (97 votes), past polls