Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re^2: Alternatives to DB for comparable lists

by cavac (Deacon)
on May 16, 2018 at 11:59 UTC ( #1214645=note: print w/replies, xml ) Need Help??

in reply to Re: Alternatives to DB for comparable lists
in thread Alternatives to DB for comparable lists

To add to your answer, i have a similar system running on some of my servers, indexing some pretty nastily-disorganized windows fileshares. I put everything into a PostgreSQL database. That lets me do all kinds of metadata analysis with a few simple SQL statements.

Everything "below a few tens of millions of entries" shouldn't be a problem for a decent low- to midrange server build within the last 8 years. My current, 8 year old, development server is used for this kind of crap all the time without any issues.

I'm pretty sure that running fstat() on all those files is going to be a major slowdown, and the checksuming certainly needs to be done locally, not over the network.

"For me, programming in Perl is like my cooking. The result may not always taste nice, but it's quick, painless and it get's food on the table."
  • Comment on Re^2: Alternatives to DB for comparable lists

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1214645]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (2)
As of 2018-07-22 01:13 GMT
Find Nodes?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?

    Results (451 votes). Check out past polls.