Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Expected speed of retrieve function (nstore retrieve)

by rjbioinf (Acolyte)
on May 16, 2012 at 11:01 UTC ( #970808=perlquestion: print w/ replies, xml ) Need Help??
rjbioinf has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I have an array stored using 'nstore' method. The resulting dump file is 90Mb (40 million elements). When I load it back into Perl using 'retrieve' it takes > 5 minutes. I have never really looked into the speed of this operation much before as I have never been left waiting as it were. I wonder if you would expect the retrieve process to take this long or is something amiss? I am running a new 4gig ram macbook pro with Lion so it should be pretty quick i would have thought? cheers,

Comment on Expected speed of retrieve function (nstore retrieve)
Re: Expected speed of retrieve function (nstore retrieve)
by Eliya (Vicar) on May 16, 2012 at 11:22 UTC
    perl -e'$x[$_]=42 for 1..40_000_000; <>'
    already needs around 1.5GB memory on my machine, and
    perl -e'$x[$_]="foo" for 1..40_000_000; <>'

    takes 3.3GB (checked with top).

    So depending on what exactly you have in your array, and whether the process of retrieving involves making another copy of the data on the stack (not that unlikely, though I haven't checked), and what else you have running on your machine, you might well be running out of physical memory...

Re: Expected speed of retrieve function (nstore retrieve)
by BrowserUk (Pope) on May 16, 2012 at 11:30 UTC
    When I load it back into Perl using 'retrieve' it takes > 5 minutes.

    That is very slow.

    Using a a 32-bit perl, nstoring a 40e6 (numeric) element array took 12 seconds and produces a 200MB file:

    C:\test>\perl32\bin\perl.exe \perl32\bin\p1.pl use Storable qw[ nstore retrieve ];; $#a = 40e6; $a[ $_ ] = $_ for 0..40e6-1;; $t=time; nstore( \@a, '/test/junk.sto' ); print time()-$t;; 12.3673732280731 Terminating on signal SIGINT(2) C:\test>dir junk.sto 16/05/2012 12:22 199,999,628 junk.sto

    Retrieving it took 12 seconds and used 1.1GB peak:

    C:\test>\perl32\bin\perl.exe \perl32\bin\p1.pl use Storable qw[ retrieve ];; $t=time; $a = retrieve( '/test/junk.sto' ); print time()-$t;; 12.2560520172119

    A couple of possibilities:

    Is the disk where you are storing it local? Heavily used? On its last legs?

    Is loading the data (briefly requiring 1.1GB) moving you into swapping?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

Re: Expected speed of retrieve function (nstore retrieve)
by Anonymous Monk on May 16, 2012 at 11:40 UTC
    For anybody else wondering, &nstore & &retrieve come from Storable module.
Re: Expected speed of retrieve function (nstore retrieve)
by sauoq (Abbot) on May 16, 2012 at 12:02 UTC
    The resulting dump file is 90Mb (40 million elements).

    What kind of elements?

    -sauoq
    "My two cents aren't worth a dime.";
      Just integer numbers. I think it is a memory issue. I shut down other memory greedy applications and the retrieve process runs faster at ~40 seconds. I think OSX Lion has problems with freeing up memory.

        There's an application for Mac OS X called "FreeMemory" which, as the name suggests, frees up memory. I've been using it for 6 months or so without any problems. It was a free download from the AppStore. You may find it useful.

        -- Ken

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://970808]
Approved by marto
Front-paged by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (6)
As of 2014-09-03 04:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (35 votes), past polls