Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re^4: Parallel::Forkmanager and large hash, running out of memory

by mabossert (Scribe)
on Apr 24, 2013 at 16:42 UTC ( [id://1030461]=note: print w/replies, xml ) Need Help??


in reply to Re^3: Parallel::Forkmanager and large hash, running out of memory
in thread Parallel::Forkmanager and large hash, running out of memory

Got it...I guess I need to lay off the crack pipe...Thanks!

Of course, this brings me to the next challenge: concurrency. Since I will be doing this work in parallel, I will have to see how SQLite handles concurrency...or perhaps lean over towards MySQL...

Thanks again for the help to you (and all others that responded...)

Replies are listed 'Best First'.
Re^5: Parallel::Forkmanager and large hash, running out of memory
by pokki (Monk) on Apr 24, 2013 at 18:37 UTC

    The relevant doc is here.

    The tl;dr: version is: any number of processes can read at the same time. When a process wants to write to the database, the engine is notified, waits for current readers to finish, then gives exclusive access to the writer process. No one can read while a write is occurring. When the write is finished, everybody can start reading again.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1030461]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others exploiting the Monastery: (5)
As of 2024-03-28 19:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found