Re: Memory Leak when slurping files in a loop (sliding window explained)

in reply to Memory Leak when slurping files in a loop

Why don't you use the sliding window technique already discussed?

            window 
|------[++++++|++++++]------|------|---|    file
   A      B      C      D      E      F     blocks
             <----->
              match
[download]

If you don't destroy/recreate the variables but just change the content, your memory consumption willš be minimal.

seek and read help reading chunks of data from files.

substr manipulates the content of strings.

pos returns the position of your last regex match.

So only one global variable $window of fixed size holding two current blocks could do and whenever the pos of a match leaves the first block you have to shift a new block into $window.

Cheers Rolf

š) well, as long as Perl doesn't do very (unlikely) weird speed optimizations.

UPDATE:

This code is an almost perfect example of what I meant: Matching in huge files

The differences are the temporary variable $block which could be optimized away and the handling of pos. Instead of adjusting the window at "halftime", pos is adjusted to the window. Actually I think this is even smarter than what I planed...

In Section Seekers of Perl Wisdom