Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

How's mldbm with concurrency?

by tphyahoo (Vicar)
on Mar 15, 2005 at 16:31 UTC ( #439668=perlquestion: print w/ replies, xml ) Need Help??
tphyahoo has asked for the wisdom of the Perl Monks concerning the following question:

I'm using MLDBM to store the results from a web crawler bot. These are all HOH values, not too complicated. I'm fairly certain that if I am just running one bot, all should be well -- though I'm watching out for the data structure memory size restriction mentioned in the bugs area of MLDBM, just hoping this won't be a problem.

But what happens if I run multiple crawlers, all writing their data to the same file? Assuming no data conflicts -- such as updating the same hash key with different data -- am I going to be okay?

If there is data conflict, does MLDBM give you feedback that there was a conflict, or just update with the first value then the second value, or what?

Data-structure wise, I don't think I need an RDBMS system. But concurrency wise?

If it seems MLDBM is going to give me trouble, can someone suggest an alternative that handles concurrency with hash serialization better?

UPDATE: Followup-ish post at Can I serialize an object and have it remember it's type?

Comment on How's mldbm with concurrency?
Re: How's mldbm with concurrency?
by jZed (Prior) on Mar 15, 2005 at 16:57 UTC
    As I understand it, MLDBM is just a way of invoking a serializer, it has nothing to do with concurrency. Using flock() or some other locking scheme on the file seems to be the way to go. One option is to use DBD::DBM (part of the DBI distribution), which provides a DBI interface to MLDBM on top of BerkeleyDB or othe DBM file and it provides flocking (on a separate lockfile, not on the DBM file) if you want it.
Re: How's mldbm with concurrency?
by perrin (Chancellor) on Mar 15, 2005 at 18:10 UTC
Re: How's mldbm with concurrency?
by merlyn (Sage) on Mar 15, 2005 at 20:23 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://439668]
Approved by thor
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (10)
As of 2014-07-22 07:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (106 votes), past polls