Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re^2: Problems with SDBM

by Laurent_R (Canon)
on Mar 15, 2013 at 14:37 UTC ( #1023712=note: print w/replies, xml ) Need Help??

in reply to Re: Problems with SDBM
in thread Problems with SDBM

Hi, thanks everyone for the answers already provided.

The main reason to tie is resource limits: the data input has about 30 million records (and slightly less than 2 GB) and that is just too large for a hash (untied hash, that is). Having said that, persistence would also be a bonus because later processes would use the same data and would not have to load it again. But persistence is not the primary reason for using tied hashes.

I am not too much concerned with speed performance at this point (although it might become important at some point, given the large data volume), my concern is that the process fails when I have loaded only about half of the data (15.8 million records), presumably because of the large volume of data. I could use several tied hashes to get around this volume limit, but that would be sort of awkward and unwieldy (and not very scalable).

It seems that the Berkeley DB is not available on our system, so it seems that it will not be an option.

Replies are listed 'Best First'.
Re^3: Problems with SDBM
by Anonymous Monk on Mar 15, 2013 at 15:11 UTC

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1023712]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (4)
As of 2018-01-22 07:18 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (232 votes). Check out past polls.