Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re^2: mem usage

by halfcountplus (Hermit)
on May 26, 2010 at 18:02 UTC ( #841770=note: print w/ replies, xml ) Need Help??


in reply to Re: mem usage
in thread mem usage

Thanks. It just barely squeezed thru -- interestingly the kernel killed firefox during the run.


Comment on Re^2: mem usage
Re^3: mem usage
by ikegami (Pope) on May 26, 2010 at 18:10 UTC
    Woops, what I posted is very buggy. Should be
    my @lines; $#lines = 12_000_000; @lines = (); <------ was missing push @lines, $_ while <IN>;
    $ perl -e'print "abcdef\n" for 1..11_000_000' | perl -E' $#a=11_000_000; @a=(); push @a, $_ while <>; say int(`ps --no-heading -o vsz $$`/1000) ' 480

    480MB for 77MB file with 11 million lines.

      Hmmm, yeah that's much better. What's the significance of this:
      @a=();
      After presizing -- I would have thought that would just clear the array?

        It does indeed clear the array, but it does not free it the underlying buffer.

        Setting $#a extends the visible size of the array, not just the internal one. The elements I proceeded to push onto the array were getting added after the 12 mil empty ones.

        @a=(); empties the visible array, but doesn't deallocate the internal buffer like undef @a; does.

        $ perl -MDevel::Peek -e' my @a = qw( a b c ); Dump(\@a,1); @a=(); Dump(\@a,1); undef(@a); Dump(\@a,1); ' SV = IV(0x816ce54) at 0x816ce58 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x817b500 FILL = 2 <---- 3 elements MAX = 3 <---- 4 slots ARYLEN = 0x0 FLAGS = (REAL) SV = IV(0x816c154) at 0x816c158 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x817b500 FILL = -1 <---- 0 elements MAX = 3 <---- 4 slots ARYLEN = 0x0 FLAGS = (REAL) SV = IV(0x816c154) at 0x816c158 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x0 <---- No array allocated FILL = -1 <---- 0 elements MAX = -1 <---- 0 slots ARYLEN = 0x0 FLAGS = (REAL)

        If you don't want to rely on that, you could do

        my @lines; $#lines = 12_000_000; $lines[$.-1] = $_ while <>; $#lines = $.-1;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://841770]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (5)
As of 2015-07-03 22:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (57 votes), past polls