Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: SHA1 sum archival methods

by redgreen (Curate)
on Nov 05, 2009 at 15:32 UTC ( #805288=note: print w/ replies, xml ) Need Help??


in reply to SHA1 sum archival methods

It sounds like overall you want to write your own database. If that is your goal, then do it.

Myself, I would use sqlite DBD::SQLite. It gives you a database with indexes, but without excessive requirements of setting up a database server. It is well tested, and fast.

It sounds like you are still in the design phase. Why not design the data storage section so it can be changed around? Start now with DBI, using SQLite, and then change it later on if needed. You would then have something to compare against, and have many options.

You say you want to share your final program. Watch out! End users will want to do more with your program than you can dream of. Just because your file archive is only 100,000 files and works great with DB_File, doesn't mean the next user with 100,000,000 files will be happy with your design. They might not mind setting up a full database for storage. Keep your options open.


Comment on Re: SHA1 sum archival methods
Re^2: SHA1 sum archival methods
by r1n0 (Beadle) on Nov 05, 2009 at 18:14 UTC
    I completely agree on leaving the design flexible for changes to another capability, later. And, I agree about sharing code. I want to thank fellow monks for the ideas. I was wondering if MLDM compares in speed to these others? Has anyone run a comparison of the speeds from these various modules in the past?

    Modules:
    DB_File
    DBD::SQLite
    MLDBM
    DBM::Deep

    I am interested in speed and ability to house a large number of records. I understand this is probably not possible, and for a large volume of records (>100,000,000) I should probably move to a full DB.

    Thank you in advance for your help.
      MLDB uses DB_File so a benchmark doesn't make any sense.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://805288]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (7)
As of 2014-12-25 17:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (161 votes), past polls