Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Parallel::Forkmanager and large hash, running out of memory

by talexb (Canon)
on Apr 24, 2013 at 15:04 UTC ( #1030432=note: print w/ replies, xml ) Need Help??


in reply to Parallel::Forkmanager and large hash, running out of memory

It seems you must be leaving out some important ionformation.

If you're converting a single TSV into an RDF, and just doing that is choking on a machine with half a terabyte of RAM, there's clearly something horribly wrong.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds


Comment on Re: Parallel::Forkmanager and large hash, running out of memory
Re^2: Parallel::Forkmanager and large hash, running out of memory
by mabossert (Sexton) on Apr 24, 2013 at 15:22 UTC

    Sorry if I was not clear enough. I am processing thousands of files. The files that contain the values that I am loading into a hash are at about 1800 files right now.

      Oh -- so you're not processing a single file at once -- that's what I took away from your OP. My mistake.

      If you're loading stuff into a hash, and *that's* overflowing memory, then it sounds like you'll need another approach, and the one that comes to mind right away is to use a database.

      Alex / talexb / Toronto

      "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1030432]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2014-04-19 18:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (483 votes), past polls