Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Oh where has my memory gone?

by Anonymous Monk
on Jun 09, 2008 at 19:03 UTC ( #691067=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I can't seem to find where all the memory is going with this script. It's not doing anything fancy in terms of coding and I'm even using undef when I'm done with variables. This script seems to end within hours of starting with "Out of Memory!" errors.
#! /usr/bin/perl -X use threads; $thr1 = threads->new(\&manager); while (1){sleep(1);} sub initiator(){ my @in_items = @_; my $torrent = substr(@in_items[0],2); chomp($torrent); my $f = substr(@in_items[1],2); chomp($f); my $lc = substr(@in_items[2],2); chomp($lc); my $priority = substr(@in_items[3],2); chomp($priority); my $buffer = substr(@in_items[4],2); chomp($buffer); my $saveas = substr(@in_items[5],2); chomp($saveas); my $a,$b,$tr_percent,$tr_time,$tr_down,$tr_up,$tr_err,$pid,$urgent +; $final = "downloader.pl $buffer $saveas $priority $f"; $status = open(PIPE,"python btdownloadheadless.py $final |") || pr +int "Couldn't initiate download"; while (<PIPE>){ #print $_; #open(STANDARD, ">>./$buffer/DEBUG.dat"); print STANDARD $_; c +lose(STANDARD); $ln = $_; chomp($ln); ($vb,$vn) = split(/\>\>\>/,$ln); ($a,$b) = split(/---> /,$_); chomp($b); if ($vn ne ""){ $pid = $vn; open(STANDARD2, ">>./$buffer/PID.d +at"); print STANDARD2 "$pid"; close(STANDARD2); } if ($a =~ "percent done"){$tr_percent = $b;} if ($a =~ "time left"){$tr_time = $b;} if ($a =~ "download rate"){$tr_down = $b;} if ($a =~ "upload rate"){$tr_up = $b;} if ($a =~ "ERROR:"){$tr_err = $_; $tr_time = "complete!";} if ($tr_time eq ""){$tr_time = "Connecting...";} open(OUT, ">>./$buffer/WATCH.dat"); print OUT "$tr_time ( % $tr_percent ) \@ $tr_down $tr_err\n"; close(OUT); if (($tr_time =~ "complete!") || ($tr_time eq "Download Succee +ded!")){ sleep(1); print "\nTrying to end process [$buffer] ($pid)...\n"; system("kill -s KILL $pid"); } $v++; } $urgent = 0; open(STANDARD, ">>./$buffer/DEBUG.dat"); print STANDARD "PIPE ENDED..."; $err = $? & 255; print STANDARD "With: $err"; print STANDARD "With: $?"; print STANDARD "With: $status"; close(PIPE); close(STANDARD); undef $a,$b,$tr_percent,$tr_time,$tr_down,$tr_up,$tr_err,$pid,$urg +ent,$v,$err,$vb,$vn,$ln; undef $torrent,$lc,$f,$priority,$buffer,$saveas,$cmd,$cancel; undef @in_items; } sub manager(){ $|++; print "."; opendir(DIR,"./acc"); @list = readdir(DIR); close(DIR); foreach (@list){ if ($_ =~ ".dat"){ @new_list[$q] = $_; $q++; }} #print @new_list; foreach (@new_list){ open(READER,"./acc/$_"); @items = <READER>; close(READER); #print @items; $thr = threads->new(\&initiator, @items); $line = $_; chomp($line); #cp("./acc/$line","./$line/FORKED.dat"); unlink("./acc/$line"); } undef @list; undef @new_list; undef @items; undef $line; $q = 0; sleep(2); &manager(); }

Replies are listed 'Best First'.
Re: Oh where has my memory gone?
by BrowserUk (Pope) on Jun 09, 2008 at 19:11 UTC
Re: Oh where has my memory gone?
by radiantmatrix (Parson) on Jun 09, 2008 at 20:27 UTC

    Well, the first answer is a good one. Let me explain it a bit. You have this code inside manager():

    sleep(2); &manager();

    Every time a subroutine calls another subroutine, the stack grows. Each subroutine in the chain hangs onto whatever memory it has. When your stack gets big enough, you'll run out of memory or blow the stack -- both are fatal.

    As an aside, you should really include use strict; use warnings; in your code -- you would have got a warning about deep recursion inside manager() long before you ran out of memory.

    If you want to avoid this, you have two options: refactor your code so that manager() is just a loop, or use goto.

    If you say goto &manager;, you're effectively1 making a jump, not a call -- your stack remains constant.

    Also, you should read perlvar, specifically the bits on my - variables that you only need inside of manager should be declared with that scope using my

    1. As shmem explains below, it's not really implemented as a jump; however, the effect is so similar that the difference is unimportant for this convsersation. Still -- thanks to shmem for the clarification.

    Updates:

    • 20080611 : altered language, see footnote 1

    <radiant.matrix>
    Ramblings and references
    “A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort.” Herm Albright
    I haven't found a problem yet that can't be solved by a well-placed trebuchet
      If you say goto &manager;, you're making a jump, not a call -- your stack remains constant.

      Not quite accurate; it's not a jump: a new stack frame is set up, the current stack frame is replaced by the new one, then unwound, so you have two of the same size. The memory used might remain constant, though. </nitpick>

      ;-)

      --shmem

      _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                    /\_¯/(q    /
      ----------------------------  \__(m.====·.(_("always off the crowd"))."·
      ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
        Its still doing it, slowly but surely this script eats all the RAM available! 0.5, 0.9, 1.4, 25... Dead! Is there a better way of writing this? It seems simple enough, but eventually the script will end with the out of memory error.
      Ahhh, I completely forgot about that. That I was simply creating multiple copies of manager (which I was). Thank you very very much, it should solve my memory issue. ;)
Re: Oh where has my memory gone?
by Narveson (Chaplain) on Jun 09, 2008 at 19:21 UTC
    I'm even using undef when I'm done with variables

    Don't bother using undef to free memory, just let your variables go out of scope and let Perl's garbage collector work for you.

    For instance, you should declare my @list and my @new_list inside the manager block.

    Followup: better yet, don't use the arrays. Just iterate over the directory handle.

    use strict; use warnings; my $thr; # ... sub manager() { local $| = 1; while (print '.') { opendir(my $dir_handle, './acc'); FILE: while (my $file = readdir($dir_handle) ) { # or use /.dat/ if that's what you meant next FILE if $file !~ /[.]dat$/; open(my $reader, '<', "./acc/$file"); my @items = <$reader>; close($reader); $thr = threads->new(\&initiator, @items); unlink("./acc/$file"); } close($dir_handle); sleep(2); } }
      Although I agree with the advice, Perl's behavior is actually more complicated. Sometimes undef does free memory.
      Hello, THANK YOU SO MUCH FOR THE QUICK REPLY AND YES I'M YELLING IT! But could I also just do...
      sub manager(){ while(1){ #same as before... } }
      To reduce memory usage? With or without the need for undef?
        Yes that'll do the same as BrowserUK's quick solution: replace the endless recursion with a loop.

        Perl doesn't treat tail recursion as speacial, so it's still bad. As in the original code.

        Yes. Lose the undefs, they are doing nothing.
Re: Oh where has my memory gone?
by toolic (Bishop) on Jun 09, 2008 at 20:23 UTC
    This is completely unrelated to your issue, but these suggestions may be useful in the future:

    Use the strictures:

    use warnings; use strict;
    Amongst other things, this will warn you that
    Scalar value @in_items[0] better written as $in_items[0]

    chomp can chomp all elements of a list at once, so there is no need to explicitly chomp each one:

    chomp (my @in_items = @_);

    It is a good practice to check the return value of open, opendir and system.

    Avoid using $a and $b since they have special meaning for sort.

Re: Oh where has my memory gone?
by perrin (Chancellor) on Jun 09, 2008 at 19:45 UTC
    Every time you make a new thread, it copies everything. If you're not on Windows, you might want to try using forking instead. It will use less memory because of copy-on-write.
      1. What about when the thread exits? Each time it initiates, they usually end a few minutes later... 2. Should I be fine now with goto / while(1) instead?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://691067]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (3)
As of 2021-05-18 10:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Perl 7 will be out ...





    Results (178 votes). Check out past polls.

    Notices?