Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re^3: Loading 283600 records (Updated)

by BrowserUk (Pope)
on Sep 23, 2012 at 16:05 UTC ( #995214=note: print w/replies, xml ) Need Help??

in reply to Re^2: Loading 283600 records (Updated)
in thread Loading 283600 records (WordNet)

There are no rules -- beyond minimising the number of opcodes called -- that apply in all situations. Try plugging this into your benchmark:

my %hash; while( <> ) { my( $k, @v ) = unpack( 'a10a10a4', $_ ); push @{ $hash{ $k } }, \@v }

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

Replies are listed 'Best First'.
Re^4: Loading 283600 records (Updated)
by remiah (Hermit) on Sep 23, 2012 at 23:10 UTC

    Added 3 test

    05 .. unpack, using List::MoreUtils's natatime
    06 .. unpack again
    07 .. yours
    Your unpack was faster than mines. This is benchmark results.
                       s/iter 02_split1 05_unpack_natatime 06_unpack_map 04_unpack 03_split2 07_unpack_2
    02_split1            6.38        --               -13%          -16%      -35%      -42%        -50%
    05_unpack_natatime   5.55       15%                 --           -4%      -25%      -33%        -43%
    06_unpack_map        5.34       19%                 4%            --      -22%      -31%        -40%
    04_unpack            4.18       53%                33%           28%        --      -11%        -24%
    03_split2            3.70       72%                50%           44%       13%        --        -14%
    07_unpack_2          3.18      100%                74%           68%       31%       16%          --
    01_substr            2.70      136%               105%           98%       55%       37%         18%
    And test code added. dsheroh told me of in memory SQLite. It's loading time is apparently faster than any of above tests. I will report it later.

Re^4: Loading 283600 records (Updated)
by remiah (Hermit) on Sep 24, 2012 at 10:31 UTC

    Hello, BrowserUK.

    I posted some benchmarks of sqlite in-memory at the bottom of this thread. It was surprise for me. Please have a look at.

    and thanks for responding to me.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://995214]
[Corion]: marioroy: Oh, that's always cool, having API-compatible modules. This makes testing and comparing things much easier
[marioroy]: IPC in MCE::Shared can handle 400k (sends) per second. That's seems a lot for being a pure-Perl module. After making the release, will come back and post a solution for a node by a fellow wanting faster logging.
[Corion]: While working on WWW::Mechanize:: Chrome, I had the suspicion that AnyEvent was doing something wrong, but I was able to swap it out for Mojolicious and the error persisted.
[Corion]: Of course, the error was in my own code ;)
[marioroy]: Corion, start and start_child in MCE::Hobo::Manager return a MCE::Hobo object, whereas P::FM returns the PID. I can have it return the PID though. I tried Hobo::Manager with several P::FM modules, just changed P::FM to MCE::Hobo::Manager and it works.

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (5)
As of 2017-05-26 08:38 GMT
Find Nodes?
    Voting Booth?