Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re^2: Parallel::Forkmanager and large hash, running out of memory

by mabossert (Scribe)
on Apr 24, 2013 at 15:25 UTC ( #1030441=note: print w/replies, xml ) Need Help??


in reply to Re: Parallel::Forkmanager and large hash, running out of memory
in thread Parallel::Forkmanager and large hash, running out of memory

Gotcha...thanks for the quick response. Will SQLite handle large sizes? I thought it had a 2GB limit...or am I smoking proverbial crack?

  • Comment on Re^2: Parallel::Forkmanager and large hash, running out of memory

Replies are listed 'Best First'.
Re^3: Parallel::Forkmanager and large hash, running out of memory
by Anonymous Monk on Apr 24, 2013 at 15:30 UTC

      Got it...I guess I need to lay off the crack pipe...Thanks!

      Of course, this brings me to the next challenge: concurrency. Since I will be doing this work in parallel, I will have to see how SQLite handles concurrency...or perhaps lean over towards MySQL...

      Thanks again for the help to you (and all others that responded...)

        The relevant doc is here.

        The tl;dr: version is: any number of processes can read at the same time. When a process wants to write to the database, the engine is notified, waits for current readers to finish, then gives exclusive access to the writer process. No one can read while a write is occurring. When the write is finished, everybody can start reading again.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1030441]
help
Chatterbox?
and a moth chases the moon...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (6)
As of 2018-07-18 20:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?















    Results (395 votes). Check out past polls.

    Notices?