Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

forks and Shared

by Anonymous Monk
on Nov 10, 2008 at 14:33 UTC ( #722663=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there, Im trying to increase the capacity of the shared variables of my webcrawler as I'm goind to index million of urls and the number of data shared will grow very fast. I'm using forks shared, but I'm looking for something which can keep the more used on memory and the less used on disk, with this in mind I looked for berkeleydb, but forksberkeleydb dont worked the way I was waiting, It always crash when I try to insert hash of hashes or array of hashes, so it's useless for my intention.
Some of you monks here a suggestion to solve my problem?


Replies are listed 'Best First'.
Re: forks and Shared
by perrin (Chancellor) on Nov 10, 2008 at 16:18 UTC
    BerkeleyDB is the fastest thing available. I'd suggest using it directly if the forks module is giving you trouble.
Re: forks and Shared
by arkturuz (Curate) on Nov 10, 2008 at 15:56 UTC
    I'm not sure this is the right solution because it's not a database-based but you could try using Cache::Memcached module (needs memcached). (There's also pretty faster version Cache::Memcached::Fast). It supports inserting nested Perl data structures, it's pretty reliable, and it works great for sharing data between processes.
Re: forks and Shared
by Anonymous Monk on Nov 10, 2008 at 16:02 UTC

      Note that MLDBM can be a pain to use correctly (see the BUGS section). I think DBM::Deep deals better with complex data structures. But it will also probably use more disk space than MLDBM. MLDBM requires deserialization + serialization steps each time you modify an element so I can imagine cases where DBM::Deep could be much faster than MLDBM.

      - tye        

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://722663]
Approved by ww
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (6)
As of 2018-06-23 12:20 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (125 votes). Check out past polls.