Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Memory Leak when slurping files in a loop

by LanX (Canon)
on Dec 07, 2010 at 11:16 UTC ( #875767=note: print w/ replies, xml ) Need Help??


in reply to Memory Leak when slurping files in a loop

Why don't you use the sliding window technique already discussed?

window |------[++++++|++++++]------|------|---| file A B C D E F blocks <-----> match

If you don't destroy/recreate the variables but just change the content, your memory consumption will╣ be minimal.

seek and read help reading chunks of data from files.

substr manipulates the content of strings.

pos returns the position of your last regex match.

So only one global variable $window of fixed size holding two current blocks could do and whenever the pos of a match leaves the first block you have to shift a new block into $window.

Cheers Rolf

╣) well, as long as Perl doesn't do very (unlikely) weird speed optimizations.

UPDATE:

This code is an almost perfect example of what I meant: Matching in huge files

The differences are the temporary variable $block which could be optimized away and the handling of pos. Instead of adjusting the window at "halftime", pos is adjusted to the window. Actually I think this is even smarter than what I planed...


Comment on Re: Memory Leak when slurping files in a loop
Select or Download Code
Re^2: Memory Leak when slurping files in a loop
by rizzy (Sexton) on Dec 07, 2010 at 16:42 UTC
    Rolf, I am planning on doing as you suggested. In the meantime (as I'm running some of the code) I thought there might be a very simple fix. Thanks!
Re^2: Memory Leak when slurping files in a loop
by rizzy (Sexton) on Dec 07, 2010 at 16:49 UTC
    By the way, kudos on your footnotes/graphic i your posts. Very helfpul!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://875767]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2014-10-25 04:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (141 votes), past polls