Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re^2: Alternatives to DB for comparable lists

by cavac (Deacon)
on May 16, 2018 at 11:59 UTC ( #1214645=note: print w/replies, xml ) Need Help??

in reply to Re: Alternatives to DB for comparable lists
in thread Alternatives to DB for comparable lists

To add to your answer, i have a similar system running on some of my servers, indexing some pretty nastily-disorganized windows fileshares. I put everything into a PostgreSQL database. That lets me do all kinds of metadata analysis with a few simple SQL statements.

Everything "below a few tens of millions of entries" shouldn't be a problem for a decent low- to midrange server build within the last 8 years. My current, 8 year old, development server is used for this kind of crap all the time without any issues.

I'm pretty sure that running fstat() on all those files is going to be a major slowdown, and the checksuming certainly needs to be done locally, not over the network.

"For me, programming in Perl is like my cooking. The result may not always taste nice, but it's quick, painless and it get's food on the table."
  • Comment on Re^2: Alternatives to DB for comparable lists

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1214645]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (5)
As of 2018-11-13 02:18 GMT
Find Nodes?
    Voting Booth?
    My code is most likely broken because:

    Results (149 votes). Check out past polls.