Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

How's mldbm with concurrency?

by tphyahoo (Vicar)
on Mar 15, 2005 at 16:31 UTC ( #439668=perlquestion: print w/ replies, xml ) Need Help??
tphyahoo has asked for the wisdom of the Perl Monks concerning the following question:

I'm using MLDBM to store the results from a web crawler bot. These are all HOH values, not too complicated. I'm fairly certain that if I am just running one bot, all should be well -- though I'm watching out for the data structure memory size restriction mentioned in the bugs area of MLDBM, just hoping this won't be a problem.

But what happens if I run multiple crawlers, all writing their data to the same file? Assuming no data conflicts -- such as updating the same hash key with different data -- am I going to be okay?

If there is data conflict, does MLDBM give you feedback that there was a conflict, or just update with the first value then the second value, or what?

Data-structure wise, I don't think I need an RDBMS system. But concurrency wise?

If it seems MLDBM is going to give me trouble, can someone suggest an alternative that handles concurrency with hash serialization better?

UPDATE: Followup-ish post at Can I serialize an object and have it remember it's type?

Replies are listed 'Best First'.
Re: How's mldbm with concurrency?
by jZed (Prior) on Mar 15, 2005 at 16:57 UTC
    As I understand it, MLDBM is just a way of invoking a serializer, it has nothing to do with concurrency. Using flock() or some other locking scheme on the file seems to be the way to go. One option is to use DBD::DBM (part of the DBI distribution), which provides a DBI interface to MLDBM on top of BerkeleyDB or othe DBM file and it provides flocking (on a separate lockfile, not on the DBM file) if you want it.
Re: How's mldbm with concurrency?
by perrin (Chancellor) on Mar 15, 2005 at 18:10 UTC
Re: How's mldbm with concurrency?
by merlyn (Sage) on Mar 15, 2005 at 20:23 UTC

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://439668]
Approved by thor
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2016-07-23 20:49 GMT
Find Nodes?
    Voting Booth?
    What is your favorite alternate name for a (specific) keyboard key?

    Results (221 votes). Check out past polls.