Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

Hi all,

a process checking DNS names unexpectedly has eaten 12GB of memory, forced swapping and ringing almost all alarm bells of system monitoring.

So I was really surprised. DBI needed 1GB to get all results from the MySQL database into the process. That was expected for some millions of rows and ok. But afterwards I wanted to check the DNS name in a fetching loop. The used process memory grew and grew what I really didn't expect in this way.

Now I reduced the whole thing to the following snippet which is more or less literally taken from the man page of Net::DNS. Combined with the very usful package Test::LeakTrace I have the following:

#!/bin/env perl use strict; use warnings; use strict; use 5.010; use local::lib './lib'; use Net::DNS; use Test::LeakTrace; die("ERROR: Please provide Loop count > 0 as first argument") unless $ +ARGV[0]; leaktrace { for (1..$ARGV[0]) { my $res = Net::DNS::Resolver->new; my @mx = mx($res, "example.com"); } } ;

The result is that Test::LeakTrace reports MANY leaks. Following the documentation of Test::LeakTrace I gave Net::DNS a chance in a way that one-shot-costs are ok but iterating over meanwhile instantiated/cached code should not need additional memory. To prove this I have a for-loop around it which can be given on the command line. Together with the following little shell script:

for iterations in 1 2 3 4 5 6 7 8 9 10; do perl script.pl $iteration +s 2>&1 | wc -l; done

I get the following output:

1374 1492 1612 1736 1859 1982 2106 2229 2353 2477

You see that the leaked memory rises with the itaration count. This seems to me very bad.

Now to you monks: I would be intersted to see the result on different plattforms with different Perl versions. If you have time and pleasure to test it on your Perl I would be really interested in the results.

By the way: If someone knows a package for DNS lookups not easting the memory of my machine should tell it. :-) Does anybody know some documentation how to read the verbose output of Test::LeakTrace (which is not used in this script)?

Thank you in advance

Regards
McA


In reply to Hunting a memory eater by McA

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others surveying the Monastery: (5)
    As of 2014-09-17 21:20 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      How do you remember the number of days in each month?











      Results (99 votes), past polls