Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Out Of Memory

by Naveen_learn (Initiate)
on Sep 18, 2013 at 09:08 UTC ( #1054621=perlquestion: print w/replies, xml ) Need Help??

Naveen_learn has asked for the wisdom of the Perl Monks concerning the following question:

I have a system which uses perl as a scripting language. When submitting input(records of half a million) my perl interpreter says "Out Of memory!"... Can you please help me on how to fix this issue?

Replies are listed 'Best First'.
Re: Out Of Memory
by choroba (Archbishop) on Sep 18, 2013 at 09:10 UTC
    Records of half a million of what? Without seeing the code and a sample of the input, we can only guess, or recommend to buy more memory.
    لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
Re: Out Of Memory
by zork42 (Monk) on Sep 18, 2013 at 09:12 UTC
Re: Out Of Memory
by Utilitarian (Vicar) on Sep 18, 2013 at 10:28 UTC
    I suspect you may be able to solve your issue by processing the individual records rather than slurping the entire input file at once, but without code and sample input we can't really provide you with more than generalisations.

    print "Good ",qw(night morning afternoon evening)[(localtime)[2]/6]," fellow monks."

      I definitely agree with this. Most of the files I am working with would give me the "out of memory" message if I tried to slurp then entirely into memory. They are just way too large. Processing one line at a time is quite often the solution. But I can't say more without seeing the program and a data sample.

Re: Out Of Memory
by roboticus (Chancellor) on Sep 18, 2013 at 11:03 UTC


    Perhaps it's time to add more memory to your machine. If you've already maxed out your memory, then maybe it's time to get a machine that can use more memory and use a 64 bit OS so you can use all the RAM you add to your machine.

    ...or you could change your program to use less RAM. But since I can't see your program, you'll have to figure that out yourself.


    When your only tool is a hammer, all problems look like your thumb.

Re: Out Of Memory
by jesuashok (Curate) on Sep 18, 2013 at 09:57 UTC
    Hi Naveen - Can you provide more information as such,
    (a) On what operating system you are running this code?
    (b)How these input records are given to program to read/parse?
    (c) Are you getting the out of memory message immediately upon executing the program or after certain process that happened and seeing it?

    Further more information would really help to provide valuable feed backs to help.
Re: Out Of Memory
by davido (Cardinal) on Sep 18, 2013 at 16:14 UTC

    I can't believe how many responses say to just add more memory. I realize this might be tongue-in-cheek, but the responsible thing for a developer to do is to figure out how to consume less memory.

    We haven't seen any code yet, nor any sample input. Thus, the question is mostly unanswerable. But attempts at providing suggestions for an unanswerable question ought to at least lean in the direction of what a developer might do to solve the problem, rather than what a hardware tech might do.

    If at all possible, read your input one record at a time, process that record, write out anything that needs to be written, accumulate whatever needs to be accumulated, and then discard from memory the current record and move on to the next. Some variation on this theme is how most of these scalability issues are resolved. Maybe you need to hold onto two or three records. Or maybe you can immediately drop any records that are uninteresting and only hold onto ones that need further work. But in the end your approach will probably look something like this.

    Another technique might be to break up the list of records into smaller chunks, and work on them individually. Accumulate results for each chunk, and in the end, compile results for all the accumulated data. This is a "map-reduce" strategy.

    So these are some very generalized suggestions, which are about as specific as possible given the ambiguities of the original question.

    To the original poster: Please consider this: When you present a question that is devoid of detail, it usually means that you have become so exacerbated with the problem that you've given up trying to figure it out yourself. But we have no chance of figuring it out either unless presented with those very details that have so frustrated you.


      I agree with your file processing strategies(++), but wanted to add an alternative:

      If you need to hold a number of inter-related records in memory, you should consider using a database - since they are designed to optimize memory usage, and provide rapid access.

                   My goal ... to kill off the slow brain cells that are holding me back from synergizing my knowledge of vertically integrated mobile platforms in local cloud-based content management system datafication.

Re: Out Of Memory
by NetWallah (Canon) on Sep 18, 2013 at 14:00 UTC
    Do you have virtual memory configured on your computer ? Can you increase it ?

    FYI - this is also known as configuring your "swap" file.

                 My goal ... to kill off the slow brain cells that are holding me back from synergizing my knowledge of vertically integrated mobile platforms in local cloud-based content management system datafication.

Re: Out Of Memory
by Anonymous Monk on Sep 18, 2013 at 16:09 UTC
    This program probably was designed to do everything in-memory such that it fails when the files grow even slightly large. Unfortunately a redesign is in order. Throwing silicon at the problem might be a stopgap .. and, if so, might be the right thing to do. Chips are cheap.

      Chips might be cheap (and the cost of labor substantially lower) but it has been my experience that such a stopgap rarely avoids the need to bite the bullet and do a proper design/redesign in the end.

Re: Out Of Memory
by Anonymous Monk on Sep 18, 2013 at 20:39 UTC
    Depends on the system and the business situation overall. This might be a mission critical hairy beast ... feed it silicon ... or it might be something much smaller ... fix it. "A system which uses Perl" sounds potentially ugly.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1054621]
Approved by marto
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (3)
As of 2021-10-26 00:34 GMT
Find Nodes?
    Voting Booth?
    My first memorable Perl project was:

    Results (90 votes). Check out past polls.