|Problems? Is your data what you think it is?|
Re: Re: Re: Re: Optimising processing for large data files.by BrowserUk (Pope)
|on Apr 10, 2004 at 23:09 UTC||Need Help??|
Okay. If anyone with perl v5.8.2 (AS 808) running under XP (or similar configuration) is following this discussion, could they please run the following code under these conditions.
Watch the 3 memory columns for perl.exe (should become the top item if you followed the above directins and don't have any other cpu intensive processes running) as the program runs.
Watch carefully, and note how the "Mem Usage" figure steadily rises for a short period before suddenly dropping back.
The "Mem Delta figure will become negative (the value displayed in braces) each time the "Mem usage" figure falls back.
Note that the "VM Size" value tracks the "Mem Usage" closely whilst being slightly larger, and grows steadily for a short period before falling back in step with "Mem Usage".
Note that each time it falls back it doesn't fall as far as it grew, resulting in an overall steady increase in the memory usage.
Note that the frequency and size of the fallbacks seems to grow ever larger, and more frequent with time.
Once you have seen enough, ^C the program.
Don't allow the "Mem Usage" value to approach the "Physical Memory Available" figure as by then you will have moved into swapping and the picture becomes confused as the OS starts swapping memory from other processors to disk and all the Mem Delta figures start showing up (negative)decreases.
I'd be really grateful if at least one other person could confirm that they too see the behaviour described.
Assuming that this behaviour isn't a figment of my imagination and is confirmed by other(s), then if anyone has a better explaination of the (temporary, but often substantial) reductions in perl.exe's memory usage, other than Perl periodically freeing heap memory back to the OS as part of some "garbage collection like" process, I'm ready to eat my hat and apologise for misleading the monks.
Examine what is said, not who speaks."Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail