Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

forks and Shared

by Anonymous Monk
on Nov 10, 2008 at 14:33 UTC ( #722663=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there, Im trying to increase the capacity of the shared variables of my webcrawler as I'm goind to index million of urls and the number of data shared will grow very fast. I'm using forks shared, but I'm looking for something which can keep the more used on memory and the less used on disk, with this in mind I looked for berkeleydb, but forksberkeleydb dont worked the way I was waiting, It always crash when I try to insert hash of hashes or array of hashes, so it's useless for my intention.
Some of you monks here a suggestion to solve my problem?

Thanks

Comment on forks and Shared
Re: forks and Shared
by arkturuz (Curate) on Nov 10, 2008 at 15:56 UTC
    I'm not sure this is the right solution because it's not a database-based but you could try using Cache::Memcached module (needs memcached). (There's also pretty faster version Cache::Memcached::Fast). It supports inserting nested Perl data structures, it's pretty reliable, and it works great for sharing data between processes.
Re: forks and Shared
by Anonymous Monk on Nov 10, 2008 at 16:02 UTC

      Note that MLDBM can be a pain to use correctly (see the BUGS section). I think DBM::Deep deals better with complex data structures. But it will also probably use more disk space than MLDBM. MLDBM requires deserialization + serialization steps each time you modify an element so I can imagine cases where DBM::Deep could be much faster than MLDBM.

      - tye        

Re: forks and Shared
by perrin (Chancellor) on Nov 10, 2008 at 16:18 UTC
    BerkeleyDB is the fastest thing available. I'd suggest using it directly if the forks module is giving you trouble.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://722663]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (8)
As of 2014-12-26 11:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (171 votes), past polls