Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Re^2: Problems with SDBM

by Laurent_R (Abbot)
on Mar 15, 2013 at 14:37 UTC ( #1023712=note: print w/replies, xml ) Need Help??

in reply to Re: Problems with SDBM
in thread Problems with SDBM

Hi, thanks everyone for the answers already provided.

The main reason to tie is resource limits: the data input has about 30 million records (and slightly less than 2 GB) and that is just too large for a hash (untied hash, that is). Having said that, persistence would also be a bonus because later processes would use the same data and would not have to load it again. But persistence is not the primary reason for using tied hashes.

I am not too much concerned with speed performance at this point (although it might become important at some point, given the large data volume), my concern is that the process fails when I have loaded only about half of the data (15.8 million records), presumably because of the large volume of data. I could use several tied hashes to get around this volume limit, but that would be sort of awkward and unwieldy (and not very scalable).

It seems that the Berkeley DB is not available on our system, so it seems that it will not be an option.

Replies are listed 'Best First'.
Re^3: Problems with SDBM
by Anonymous Monk on Mar 15, 2013 at 15:11 UTC

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1023712]
and the fog begins to lift...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2017-08-22 21:23 GMT
Find Nodes?
    Voting Booth?
    Who is your favorite scientist and why?

    Results (342 votes). Check out past polls.