Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: warning "Out of memory!"

by sundialsvc4 (Monsignor)
on Jan 23, 2014 at 14:45 UTC ( #1071766=note: print w/ replies, xml ) Need Help??


in reply to warning "Out of memory!"

Nevertheless, it is easy enough to “eyeball” the practices that are being used here and to guess what the problem is:   you are “slurping” probably many megabytes of data into memory and processing it as arrays.   Maybe it’s another one of the files that is causing the problem, or something in combination.   In any case, a 32-bit machine on a very good day will allow only a memory-space of 2gb+ for everything that a process can address, and that fills up fast.   I presume that this machine has all the RAM, physically installed now, that its motherboard is capable of.   (On a 64-bit system, the theoretical limit is much larger, but virtual-memory paging can still bring you to your knees.)

If it is simply an input file that is too-large, you can consume that file one record at a time.   If several of the files are very large, you might have to conceive of a way to process the data in multiple stages ... multiple runs through the essential algorithm, with a slice of the inputs each time.   Or, other ways ... sorting the data to take advantage of the grouping that this gives; storing data (say) in an SQLite database file so that you can, by means of a query, retrieve from an arbitrarily-large source file only the data which might match.   And so on.   The width of the memory-address, and the installed-RAM capacity, is an absolutely “hard” limitation.


Comment on Re: warning "Out of memory!"

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1071766]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (7)
As of 2014-07-30 00:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (229 votes), past polls