Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: mem usage

by almut (Canon)
on May 26, 2010 at 17:47 UTC ( #841767=note: print w/replies, xml ) Need Help??

in reply to mem usage

Perl data structures (see PerlGuts Illustrated) need more memory than the mere user data they hold.

A quick check shows that reading 1_000_000 empty lines into an array (the way you do it) leads to a process size of 192MB on my machine (with Perl 5.10.1).  So if you have 11_000_000 non-empty ones...

Update: using ikegami's suggestion, the memory usage for the same 1_000_000 empty lines reduces to 78MB without pre-extending, and (interestingly) 86MB with pre-extending the array ($#lines = 1_000_000).   Update2: and 93MB with just $#lines = 100_000 (??)

Update3: with @lines = (), pre-extending the array no longer increases the memory requirements. But it doesn't help to reduce it either (presumably, it just improves speed).

Replies are listed 'Best First'.
Re^2: mem usage
by ikegami (Pope) on May 26, 2010 at 18:12 UTC
    There was a bug causing the array to have twice the desired number of elements. I don't know if you were affected by it.
Re^2: mem usage
by halfcountplus (Hermit) on May 26, 2010 at 18:05 UTC
    Perl data structures (see PerlGuts Illustrated) need more memory than the mere user data they hold.

    Will read. I presumed they used a bit more space, say a few pointers per element for an array -- but that is really really quite a lot.

    I've also read somewhere that perl allocates in "power of 2" chunks. So when you add a 2nd element, you get allocated space for 4, then at 5 you get 8, at 9 you get 16 and so on. Is this true?

      Close. When it grows, the array size doubles in size plus 4. An array with 1000 scalars in it could take up as much as

      Array overhead + ( 999*2+4 ) * size of a pointer + 1000 * size of the scalars

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://841767]
vrk gets a tall glass of milk to dunk cookies.

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (8)
As of 2017-05-30 08:35 GMT
Find Nodes?
    Voting Booth?