Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Enforce Memory Cleanup

by solegaonkar (Beadle)
on Jun 17, 2014 at 05:05 UTC ( #1090101=perlquestion: print w/ replies, xml ) Need Help??
solegaonkar has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks I have a strange problem here. I read a huge file (1+ GB) 1000 records at a time. I read the records into hash map, process them and then try to free the memory using undef. Then process the next 1000 records and so on. But, strangely the code runs out of memory after some time. Looks like the garbage collection is not triggered in time. Is there some way I can enforce garbage collection? Have you faced such a problem? The code is something like this...
my %map while (sysread(CSV, $record, 66)) { $map{substr($record, 18, 14)}->{substr($record, 3, 15)} = subs +tr($record, 36, 29); if ($count++ > 1000) { &process(); undef %map; } }
Update:: The problem was solved by using scope rather than delete/undef... Updated the code to the following and that performs much better!!
while (sysread(CSV, $record, 66)) { my %map my $count=0; $map{substr($record, 18, 14)}->{substr($record, 3, 15)} = substr($ +record, 36, 29); while (sysread(CSV, $record, 66)) { $map{substr($record, 18, 14)}->{substr($record, 3, 15)} = subs +tr($record, 36, 29); if ($count++ > 1000) { &process(\%map); last; } } &process }
Thank you all for your help!!

Comment on Enforce Memory Cleanup
Select or Download Code
Re: Enforce Memory Cleanup
by Anonymous Monk on Jun 17, 2014 at 06:37 UTC
      Thanks for your reply, but I don't see any possible memory leakage here. I have attached the code above.. This one blows up with an empty process method ...

        Thanks for your reply, but I don't see any possible memory leakage here. I have attached the code above.. This one blows up with an empty process method ...

        Well, in that code, %tmap keeps growing ; %tmap keeps eating memory ; until %tmap has eaten all the memory; does that make sense to you?

        ... $tmap ... &process(); ...

        That looks suspiciously like you're not coping with scoping, subs should take arguments ...

Re: Enforce Memory Cleanup
by DrHyde (Prior) on Jun 17, 2014 at 11:42 UTC
    I suggest that you reduce your code to the minimum possible that demonstrates the problem. In doing so you might solve the problem yourself, but if not then you'll have something useful to show us and then we can help you. You will also, of course, need to show us some sample data as well.
      Thanks for your reply. I have attached the code above.. This one blows up with an empty process method. Can you see any possible problem here? Do I need to take special care because it is a hash of hash?
Re: Enforce Memory Cleanup
by Theodore (Scribe) on Jun 17, 2014 at 15:16 UTC
    undefining a hash key will not remove its key, deleteing will do that.
      Hi delete does clear the memory... but it takes ages to complete - since the code has to look through each key in the hash... We need some faster way to clear the memory
Re: Enforce Memory Cleanup
by Laurent_R (Parson) on Jun 17, 2014 at 18:01 UTC
    Are you undefining the hash or the hash values? Deleting the keys would probably work, but I suspect that a proper use of scope would probably be faster at freeing the memory used by the hash. You should really show us your code for a better comprehension of what is happening and what to do.
      Guess you are right.. Am I messing up things because of the scope here? Not very sure. I have added the code above. Can you please have a look at that - to see if there is something I am missing there?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1090101]
Approved by kevbot
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2014-09-20 18:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (160 votes), past polls