Beefy Boxes and Bandwidth Generously Provided by pair Networks Cowboy Neal with Hat
Perl: the Markov chain saw
 
PerlMonks  

Re^2: mem usage

by halfcountplus (Hermit)
on May 26, 2010 at 18:02 UTC ( #841770=note: print w/ replies, xml ) Need Help??


in reply to Re: mem usage
in thread mem usage

Thanks. It just barely squeezed thru -- interestingly the kernel killed firefox during the run.


Comment on Re^2: mem usage
Re^3: mem usage
by ikegami (Pope) on May 26, 2010 at 18:10 UTC
    Woops, what I posted is very buggy. Should be
    my @lines; $#lines = 12_000_000; @lines = (); <------ was missing push @lines, $_ while <IN>;
    $ perl -e'print "abcdef\n" for 1..11_000_000' | perl -E' $#a=11_000_000; @a=(); push @a, $_ while <>; say int(`ps --no-heading -o vsz $$`/1000) ' 480

    480MB for 77MB file with 11 million lines.

      Hmmm, yeah that's much better. What's the significance of this:
      @a=();
      After presizing -- I would have thought that would just clear the array?

        It does indeed clear the array, but it does not free it the underlying buffer.

        Setting $#a extends the visible size of the array, not just the internal one. The elements I proceeded to push onto the array were getting added after the 12 mil empty ones.

        @a=(); empties the visible array, but doesn't deallocate the internal buffer like undef @a; does.

        $ perl -MDevel::Peek -e' my @a = qw( a b c ); Dump(\@a,1); @a=(); Dump(\@a,1); undef(@a); Dump(\@a,1); ' SV = IV(0x816ce54) at 0x816ce58 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x817b500 FILL = 2 <---- 3 elements MAX = 3 <---- 4 slots ARYLEN = 0x0 FLAGS = (REAL) SV = IV(0x816c154) at 0x816c158 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x817b500 FILL = -1 <---- 0 elements MAX = 3 <---- 4 slots ARYLEN = 0x0 FLAGS = (REAL) SV = IV(0x816c154) at 0x816c158 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x817bc50 SV = PVAV(0x816d038) at 0x817bc50 REFCNT = 2 FLAGS = (PADMY) ARRAY = 0x0 <---- No array allocated FILL = -1 <---- 0 elements MAX = -1 <---- 0 slots ARYLEN = 0x0 FLAGS = (REAL)

        If you don't want to rely on that, you could do

        my @lines; $#lines = 12_000_000; $lines[$.-1] = $_ while <>; $#lines = $.-1;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://841770]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (16)
As of 2014-04-16 20:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (434 votes), past polls