Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^3: mem usage

by ikegami (Pope)
on May 26, 2010 at 18:10 UTC ( #841776=note: print w/ replies, xml ) Need Help??


in reply to Re^2: mem usage
in thread mem usage

Woops, what I posted is very buggy. Should be

my @lines; $#lines = 12_000_000; @lines = (); <------ was missing push @lines, $_ while <IN>;
$ perl -e'print "abcdef\n" for 1..11_000_000' | perl -E' $#a=11_000_000; @a=(); push @a, $_ while <>; say int(`ps --no-heading -o vsz $$`/1000) ' 480

480MB for 77MB file with 11 million lines.


Comment on Re^3: mem usage
Select or Download Code
Re^4: mem usage
by halfcountplus (Hermit) on May 26, 2010 at 18:40 UTC
    Hmmm, yeah that's much better. What's the significance of this:
    @a=();
    After presizing -- I would have thought that would just clear the array?

      It does indeed clear the array, but it does not free it the underlying buffer.

      Setting $#a extends the visible size of the array, not just the internal one. The elements I proceeded to push onto the array were getting added after the 12 mil empty ones.

      @a=(); empties the visible array, but doesn't deallocate the internal buffer like undef @a; does.

      $ perl -MDevel::Peek -e' my @a = qw( a b c ); Dump(\@a,1); @a=(); Dump(\@a,1); undef(@a); Dump(\@a,1); ' SV = IV(0x816ce54) at 0x816ce58 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x817b500 FILL = 2 <---- 3 elements MAX = 3 <---- 4 slots ARYLEN = 0x0 FLAGS = (REAL) SV = IV(0x816c154) at 0x816c158 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x817b500 FILL = -1 <---- 0 elements MAX = 3 <---- 4 slots ARYLEN = 0x0 FLAGS = (REAL) SV = IV(0x816c154) at 0x816c158 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x0 <---- No array allocated FILL = -1 <---- 0 elements MAX = -1 <---- 0 slots ARYLEN = 0x0 FLAGS = (REAL)

      If you don't want to rely on that, you could do

      my @lines; $#lines = 12_000_000; $lines[$.-1] = $_ while <>; $#lines = $.-1;
        Devel::Peek looks interesting. Thanks again.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://841776]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (14)
As of 2015-07-02 21:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (45 votes), past polls