Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re: warning "Out of memory!"

by sundialsvc4 (Abbot)
on Jan 23, 2014 at 14:45 UTC ( #1071766=note: print w/replies, xml ) Need Help??

in reply to warning "Out of memory!"

Nevertheless, it is easy enough to “eyeball” the practices that are being used here and to guess what the problem is:   you are “slurping” probably many megabytes of data into memory and processing it as arrays.   Maybe it’s another one of the files that is causing the problem, or something in combination.   In any case, a 32-bit machine on a very good day will allow only a memory-space of 2gb+ for everything that a process can address, and that fills up fast.   I presume that this machine has all the RAM, physically installed now, that its motherboard is capable of.   (On a 64-bit system, the theoretical limit is much larger, but virtual-memory paging can still bring you to your knees.)

If it is simply an input file that is too-large, you can consume that file one record at a time.   If several of the files are very large, you might have to conceive of a way to process the data in multiple stages ... multiple runs through the essential algorithm, with a slice of the inputs each time.   Or, other ways ... sorting the data to take advantage of the grouping that this gives; storing data (say) in an SQLite database file so that you can, by means of a query, retrieve from an arbitrarily-large source file only the data which might match.   And so on.   The width of the memory-address, and the installed-RAM capacity, is an absolutely “hard” limitation.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1071766]
[james28909]: politicians making jobs for themselves? that is absurd! we all know how hard those politicians have it, having to take millions from lobbiest while screwing over their citizens. have a little sympathy why dont ya!

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (9)
As of 2017-12-15 18:33 GMT
Find Nodes?
    Voting Booth?
    What programming language do you hate the most?

    Results (442 votes). Check out past polls.