Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Looking for Leaks in all the wrong places

by Stegalex (Chaplain)
on Jun 09, 2002 at 19:37 UTC ( [id://172946]=perlquestion: print w/replies, xml ) Need Help??

Stegalex has asked for the wisdom of the Perl Monks concerning the following question:

I am convinced that there is a memory leak somewhere in my code. I would like to ferret it out. I noticed that there are two modules Devel::Leak and Apache::Leak that look like they might be helpful in pinpointing the problem. Unfortunately I have been unable to uncover much documentation about them.

Has anyone used these modules? If so, please post some sample code and an explanation of how to use them properly. Thanks!

~~~~~~~~~~~~~~~
I like chicken.

Replies are listed 'Best First'.
Re: Looking for Leaks in all the wrong places
by Matts (Deacon) on Jun 09, 2002 at 20:21 UTC
    This is a tough one. Finding memory leaks in Perl code is *really* hard. Unfortunately Devel::Leak (and Apache::Leak which is based on the former) are next to no use whatsoever, because they tend to tell you only that you have a leak, not what caused it.

    You may find some wisdom in the mod_perl guide section on Debugging. But I wouldn't hold out too much hope there.

    My personal solution to detecting memory leaks in mod_perl scripts (and which I used with good success in finding memory leaks in AxKit, is to use what I guess is a simple step through system, with a slight difference.

    First of all it's important to note how AxKit is structured (and try and apply it to your code if you can).

    In AxKit, the entire request is wrapped in an eval{} block. At the end of the eval block if I detect an error I can handle it appropriately, depending on what kind of error the user has and what configuration is set. But don't worry too much about that - the important thing is the eval{} block. Even if you don't write the error handling bit, you can debug this just by using that eval{} block and ignoring that your script might return something invalid for now.

    Now in order to debug my memory leaks, I first find a URI that causes the leak on every request. Then I set apache running in single thread mode (/path/to/apache/bin/httpd -X). Then I hit the page with apachebench (/path/to/apache/bin/ab). I check the process is leaking by watching it in another window running "top". Following so far? Good.

    Now, I add a simple: die "Leak test"; to the start of my eval{} block (just inside the opening curly bracket). Stop/start apache (still in single thread mode). And hit it again with apachebench. Your request should not leak here. If it does, you know the problem is in your eval handling code or before you got to your die().

    Now move your "die" further into your code, following the path you know this request takes. Use fairly coarse steps at first, and when you get into the leak, step back in single lines.

    I guarantee you that eventually you will discover where your leak is.

    Now of course fixing it, that's another matter... The things that are the worst culprits for memory leaks in perl code are:

    • Closures - this is a particular concern for mod_perl since the closure will sometimes remain active for the lifetime of the process. Fixing closure related memory leaks requires an understanding of what a closure is and what it's for and how and why it holds onto an instance of your variable.
    • Circular references - these nasty little buggers mean your variables never get freed. There are techniques for avoiding them, and almost always they are unnecessary.
    • Perl bugs - yes there are such things. Like for example in 5.00503, the following code (which really is a think-o on my part for writing it in the first place as it makes no real sense) leaked like a seive:
      my @foo = get_list() if $condition;
      (or it was something like it - I don't fully remember the exact details now, they are lost in the anals of axkit history)
    • XS modules - sometimes an author of an XS module will forget to free some memory, or mess up the refcount of a variable. It's very easy to do - I should know ;-)

    I hope that's some help. I feel deep sympathy for you as finding memory leaks is no fun at all. Good luck!

      It should be noted that closures fall into the perl bugs section. They leak no matter what you do. Perl through 5.6.1 has some leaks that there's just not much you can do about, unfortunately, short of avoiding the conditions that cause them.

      5.8.0 should be much better in this regard, though it's not quite out yet.

      Thanks Matts! And hey, I really enjoy using all your XML stuff too.

      The process for identifying memory leaks looks like a real pain in the butt. After thinking about it for a while, I would be perfectly happy to just automatically reboot the server after CPU / Memory are depleted to a certain point.

      So my new question is: What combination of CPU utilization / memory paucity should I use as the trigger to automatically reboot?

      My system is a single processor Linux box with 750M of memory running Apache and Oracle with mod_perl.

      ~~~~~~~~~~~~~~~
      I like chicken.
        The process for identifying memory leaks looks like a real pain in the butt. After thinking about it for a while, I would be perfectly happy to just automatically reboot the server after CPU / Memory are depleted to a certain point.
        this is a really bad attitude to take towards your code. not only has matts given you great code which you compliment, but he's given you an outline for advanced debugging techniques he used in the same code you praise.

        instead of taking the opportunity to learn more about bugs that might be in your code (and software development in general) from someone who took the time to provide an detailed answer your question, you've decided to throw away his proven advice.

        if you feel his advice is over your head, and you would like to learn but don't think you're capable yet, say so. i'm sure other suggestions can be offered, including advice on your follow-up question. as it stands, your dismissive tone leaves me unsympathetic to your current situation.

        why would anyone here want to provide you with help in the future if you openly throw away such good advice?

        i certainly wouldn't.

        ~Particle *accelerates*

        A reply falls below the community's threshold of quality. You may see it by logging in.
        You should *never* need to reboot the whole server - instead set MaxRequestsPerChild low (axkit.org has it at 100), and install something like Apache::SizeLimit to make sure you don't get one particular process going totally out of control.
        I agree with particle on this one. Who is to say that tomorrow you won't write another script with exactly the same problem your current one has? In that case, your server resources will become depleted twice as fast without an end in sight. Also, why be satisfied with code that you know is broken?

        thor

Re: Looking for Leaks in all the wrong places
by gav^ (Curate) on Jun 09, 2002 at 23:56 UTC
    If you are running Apache, you may find it nice to have a cron job that restarts it every 24 hours or so.

    I've always did this because it sounded like a sensible thing to do, but it wasn't until recently I found out why :) We had a server which apache was never restarted because it sometimes forked a long running child process that would also be killed. As a "quick fix" the cronjob was turned off. Weeks later, the machine was out of memory and wierd things started happening due to it having 8 Apache child process each using 25-27 meg of memory.

    Often debugging a memory leak isn't a viable option, the main use of this box was to run an app which is a big ball of mud and contains > 7,000 lines of frightening perl.

    gav^

      Better yet, use Apache::SizeLimit which can handle this problem automatically. Adam Robinson just added support for Apache::SizeLimit to Bricolage and the results have been impressive.

      -sam

        Yes, this is a perfect job for Apache::SizeLimit. And I'm not just saying that because I'm the maintainer. No one should run mod_perl without it (or its cousin Apache::GTopLimit).

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://172946]
Approved by Stegalex
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2024-03-28 17:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found