Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Perl out of memory

by Anonymous Monk
on Feb 29, 2012 at 17:52 UTC ( #956984=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi - i've been using this perl script for quite some time to pull data from a directory, I have limited perl knowledge and recently the directory has grown very big. It is big enough that I now get a 'out of memory!' error when running the script in active state perl on my windows xp desktop. I've posted the code below...does anyone know how to fix this?

if ( $mesg->code ) { # Handle error codes here print ("There was a bind error\n"); } $mesg = $ldap->search( # perform a search base => "ldap base here", filter => "ldap filter here", attrs => ['ldap attributes here'] ); $mesg->code && die $mesg->error; foreach $entry ($mesg->entries) { $entry->dump; } $mesg = $ldap->unbind; # take down session

Replies are listed 'Best First'.
Re: Perl out of memory
by atcroft (Abbot) on Feb 29, 2012 at 19:36 UTC

    The issue you are running into is because you are collecting all of the matching LDAP results before processing, rather than processing them one at a time. Depending on the system, this could easily be a number of results which would cause you to run out of memory. You will probably want to look at using the callback option on the search method, which should help you to do so.

    Hope that helps.

Re: Perl out of memory
by GrandFather (Sage) on Feb 29, 2012 at 19:31 UTC

    What makes you think the problem is in the code fragment you showed? Can you add about 10 lines of code to make that a runnable script that shows the problem? How big is "very big"?

    True laziness is hard work

      Here are the first lines

      use Net::LDAP; $ldap = Net::LDAP->new('ip address') or die("Could not get to server") +; # bind to a directory with dn and password $mesg = $ldap->bind( 'user dn', password => 'password' );

      the problem is I really don't know how to code in perl, someone wrote this script and left it with me. I appreciate the suggestions but I am not familiar with some of the things you guys have mentioned. This ends up exporting a 254mb file and gets out of memory. Its 160k users with attributes.

Re: Perl out of memory
by nikosv (Chaplain) on Feb 29, 2012 at 20:14 UTC
    In cases like that it is best to use an iterator so you get one item back at a time and not fill memory at once with all the items.In other languages (like in C#) this happens by using 'yield' typically with a foreach loop,while in Perl the concept is explained in MJ Dominus's HOP book

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://956984]
Approved by herveus
and the pool shimmers...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (2)
As of 2018-03-20 04:54 GMT
Find Nodes?
    Voting Booth?
    When I think of a mole I think of:

    Results (247 votes). Check out past polls.