Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: mem usage

by ikegami (Pope)
on May 26, 2010 at 17:34 UTC ( #841761=note: print w/replies, xml ) Need Help??


in reply to mem usage

Maybe fragmentation from extending the stack is using up a lot of memory. Try to avoid putting the entire file on the stack, and pre-extending its destination:

my @lines; $#lines = 12_000_000; @lines = (); push @lines, $_ while <IN>;

Note that shuffle already exists in List::Util.

Update: Fixed bug.

Replies are listed 'Best First'.
Re^2: mem usage
by halfcountplus (Hermit) on May 26, 2010 at 18:02 UTC
    Thanks. It just barely squeezed thru -- interestingly the kernel killed firefox during the run.
      Woops, what I posted is very buggy. Should be
      my @lines; $#lines = 12_000_000; @lines = (); <------ was missing push @lines, $_ while <IN>;
      $ perl -e'print "abcdef\n" for 1..11_000_000' | perl -E' $#a=11_000_000; @a=(); push @a, $_ while <>; say int(`ps --no-heading -o vsz $$`/1000) ' 480

      480MB for 77MB file with 11 million lines.

        Hmmm, yeah that's much better. What's the significance of this:
        @a=();
        After presizing -- I would have thought that would just clear the array?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://841761]
help
Chatterbox?
[Corion]: Nicholas Clark++ # read for a multichar block-delimiter by scanning for the last char, and only then check whether the whole delimiter was read.
[Corion]: At least if you're appending the data read to a larger buffer, this means you avoid the situation of "a delimiter was found but the other half has not been read yet"
choroba misses the context
[Corion]: choroba: Optimizing how Perl reads source code on startup (and then processes it line-by-line)
[Corion]: But in general, it seems to be an interesting approach I should think about - whenever I'm searching for something, to consider if I could search for the end of the token instead of the start of the token

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (11)
As of 2016-12-06 15:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:













    Results (108 votes). Check out past polls.