Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Memory Leak when slurping files in a loop

by LanX (Canon)
on Dec 07, 2010 at 11:16 UTC ( #875767=note: print w/ replies, xml ) Need Help??


in reply to Memory Leak when slurping files in a loop

Why don't you use the sliding window technique already discussed?

window |------[++++++|++++++]------|------|---| file A B C D E F blocks <-----> match

If you don't destroy/recreate the variables but just change the content, your memory consumption will╣ be minimal.

seek and read help reading chunks of data from files.

substr manipulates the content of strings.

pos returns the position of your last regex match.

So only one global variable $window of fixed size holding two current blocks could do and whenever the pos of a match leaves the first block you have to shift a new block into $window.

Cheers Rolf

╣) well, as long as Perl doesn't do very (unlikely) weird speed optimizations.

UPDATE:

This code is an almost perfect example of what I meant: Matching in huge files

The differences are the temporary variable $block which could be optimized away and the handling of pos. Instead of adjusting the window at "halftime", pos is adjusted to the window. Actually I think this is even smarter than what I planed...


Comment on Re: Memory Leak when slurping files in a loop
Select or Download Code
Re^2: Memory Leak when slurping files in a loop
by rizzy (Sexton) on Dec 07, 2010 at 16:42 UTC
    Rolf, I am planning on doing as you suggested. In the meantime (as I'm running some of the code) I thought there might be a very simple fix. Thanks!
Re^2: Memory Leak when slurping files in a loop
by rizzy (Sexton) on Dec 07, 2010 at 16:49 UTC
    By the way, kudos on your footnotes/graphic i your posts. Very helfpul!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://875767]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (9)
As of 2015-07-03 17:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (54 votes), past polls