Re: Memory Leak when slurping files in a loop (sliding window explained)

in reply to Memory Leak when slurping files in a loop

Why don't you use the sliding window technique already discussed?

            window 
|------[++++++|++++++]------|------|---|    file
   A      B      C      D      E      F     blocks
             <----->
              match
[download]

If you don't destroy/recreate the variables but just change the content, your memory consumption willš be minimal.

seek and read help reading chunks of data from files.

substr manipulates the content of strings.

pos returns the position of your last regex match.

So only one global variable $window of fixed size holding two current blocks could do and whenever the pos of a match leaves the first block you have to shift a new block into $window.

Cheers Rolf

š) well, as long as Perl doesn't do very (unlikely) weird speed optimizations.

UPDATE:

This code is an almost perfect example of what I meant: Matching in huge files

The differences are the temporary variable $block which could be optimized away and the handling of pos. Instead of adjusting the window at "halftime", pos is adjusted to the window. Actually I think this is even smarter than what I planed...

Comment on Re: Memory Leak when slurping files in a loop (sliding window explained) Select or Download Code

Replies are listed 'Best First'.
Re^2: Memory Leak when slurping files in a loop by rizzy (Sexton) on Dec 07, 2010 at 16:49 UTC
By the way, kudos on your footnotes/graphic i your posts. Very helfpul!	[reply]
Re^2: Memory Leak when slurping files in a loop by rizzy (Sexton) on Dec 07, 2010 at 16:42 UTC
Rolf, I am planning on doing as you suggested. In the meantime (as I'm running some of the code) I thought there might be a very simple fix. Thanks!	[reply]

In Section Seekers of Perl Wisdom