Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Parallel::Forkmanager and large hash, running out of memory

by talexb (Canon)
on Apr 24, 2013 at 15:04 UTC ( #1030432=note: print w/ replies, xml ) Need Help??


in reply to Parallel::Forkmanager and large hash, running out of memory

It seems you must be leaving out some important ionformation.

If you're converting a single TSV into an RDF, and just doing that is choking on a machine with half a terabyte of RAM, there's clearly something horribly wrong.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds


Comment on Re: Parallel::Forkmanager and large hash, running out of memory
Re^2: Parallel::Forkmanager and large hash, running out of memory
by mabossert (Beadle) on Apr 24, 2013 at 15:22 UTC

    Sorry if I was not clear enough. I am processing thousands of files. The files that contain the values that I am loading into a hash are at about 1800 files right now.

      Oh -- so you're not processing a single file at once -- that's what I took away from your OP. My mistake.

      If you're loading stuff into a hash, and *that's* overflowing memory, then it sounds like you'll need another approach, and the one that comes to mind right away is to use a database.

      Alex / talexb / Toronto

      "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1030432]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (6)
As of 2015-07-07 02:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (86 votes), past polls