Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re: (OT) should i limit number of files in a directory

by RMGir (Prior)
on Sep 11, 2008 at 16:13 UTC ( [id://710670]=note: print w/replies, xml ) Need Help??


in reply to (OT) should i limit number of files in a directory

merlyn's making a good point. This really sounds like a job for a database.

But if you MUST use the filesystem, then yes, you'll definitely need to do something multilevel. Any operations on directories tend to suffer badly when the file count gets high, and "high" in this context is on the order of 10,000's, not millions.

I'd strongly suggest NOT doing "op/us/cows" "optimization". If the full filename is in the leaves, a lot of operations get simpler (since you don't need to remember the path to the file).

And if your filesystem ever got corrupted, you'd never be able to recover - you might be left with an orphaned directory full of files named "cows","goats",etc... with no way of knowing that they belong under op/us. With full filenames, you can survive "mid-tree" corruption without issues, assuming fsck rescues the orphaned data.


Mike

Replies are listed 'Best First'.
Re^2: (OT) should i limit number of files in a directory
by leocharre (Priest) on Sep 11, 2008 at 16:43 UTC

    Thank you for the advice on not storing '27f1f49c9d06b5725abff58587d68b05' as '27/f1/f49c9d06b5725abff58587d68b05' - It was a cute and clever idea- but then so was seeing what would happen if I stuck two copper wires into an electric socket when I was 3 years old.

    Very helpful insight- really kept me from doing something stupid! Thank you!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://710670]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2024-03-19 11:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found