Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Expected speed of retrieve function (nstore retrieve)

by rjbioinf (Acolyte)
on May 16, 2012 at 11:01 UTC ( #970808=perlquestion: print w/replies, xml ) Need Help??
rjbioinf has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I have an array stored using 'nstore' method. The resulting dump file is 90Mb (40 million elements). When I load it back into Perl using 'retrieve' it takes > 5 minutes. I have never really looked into the speed of this operation much before as I have never been left waiting as it were. I wonder if you would expect the retrieve process to take this long or is something amiss? I am running a new 4gig ram macbook pro with Lion so it should be pretty quick i would have thought? cheers,
  • Comment on Expected speed of retrieve function (nstore retrieve)

Replies are listed 'Best First'.
Re: Expected speed of retrieve function (nstore retrieve)
by BrowserUk (Pope) on May 16, 2012 at 11:30 UTC
    When I load it back into Perl using 'retrieve' it takes > 5 minutes.

    That is very slow.

    Using a a 32-bit perl, nstoring a 40e6 (numeric) element array took 12 seconds and produces a 200MB file:

    C:\test>\perl32\bin\perl.exe \perl32\bin\ use Storable qw[ nstore retrieve ];; $#a = 40e6; $a[ $_ ] = $_ for 0..40e6-1;; $t=time; nstore( \@a, '/test/junk.sto' ); print time()-$t;; 12.3673732280731 Terminating on signal SIGINT(2) C:\test>dir junk.sto 16/05/2012 12:22 199,999,628 junk.sto

    Retrieving it took 12 seconds and used 1.1GB peak:

    C:\test>\perl32\bin\perl.exe \perl32\bin\ use Storable qw[ retrieve ];; $t=time; $a = retrieve( '/test/junk.sto' ); print time()-$t;; 12.2560520172119

    A couple of possibilities:

    Is the disk where you are storing it local? Heavily used? On its last legs?

    Is loading the data (briefly requiring 1.1GB) moving you into swapping?

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

Re: Expected speed of retrieve function (nstore retrieve)
by Eliya (Vicar) on May 16, 2012 at 11:22 UTC
    perl -e'$x[$_]=42 for 1..40_000_000; <>'
    already needs around 1.5GB memory on my machine, and
    perl -e'$x[$_]="foo" for 1..40_000_000; <>'

    takes 3.3GB (checked with top).

    So depending on what exactly you have in your array, and whether the process of retrieving involves making another copy of the data on the stack (not that unlikely, though I haven't checked), and what else you have running on your machine, you might well be running out of physical memory...

Re: Expected speed of retrieve function (nstore retrieve)
by Anonymous Monk on May 16, 2012 at 11:40 UTC
    For anybody else wondering, &nstore & &retrieve come from Storable module.
Re: Expected speed of retrieve function (nstore retrieve)
by sauoq (Abbot) on May 16, 2012 at 12:02 UTC
    The resulting dump file is 90Mb (40 million elements).

    What kind of elements?

    "My two cents aren't worth a dime.";
      Just integer numbers. I think it is a memory issue. I shut down other memory greedy applications and the retrieve process runs faster at ~40 seconds. I think OSX Lion has problems with freeing up memory.

        There's an application for Mac OS X called "FreeMemory" which, as the name suggests, frees up memory. I've been using it for 6 months or so without any problems. It was a free download from the AppStore. You may find it useful.

        -- Ken

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://970808]
Approved by marto
Front-paged by marto
[Cosmic37]: now I have tried another blunder - can anyone explain why I am such a dunderhead?
[Cosmic37]: if ( $line =~ /$mydt/ ) { print $line; }
[Cosmic37]: I try to match successive date times stored in variable $mydt
[Cosmic37]: I guess it is searching for the string "$mydt"
[Corion]: Indeed cool, erix ;)
[Cosmic37]: rather than the value of $mydt which is a date time strong such as 2016-01-01 12:30:56
[Corion]: Cosmic37: No, but maybe $mydt doesn't contain what you think it does, or it contains characters that are special in a regular expression? Try if( $line =~ /\Q$mydt\E/) { ... for a literal match
[Cosmic37]: I mean string grrr
[Corion]: Maybe add an else branch in which you print what the values of $line and $mydt are?
[Cosmic37]: ah thank you I will try

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (8)
As of 2017-06-29 17:00 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (673 votes). Check out past polls.