Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^4: Store a huge amount of data on disk

by Sewi (Friar)
on Oct 19, 2011 at 05:13 UTC ( #932288=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Store a huge amount of data on disk
in thread Store a huge amount of data on disk

Thanks, I used that solution for a number of issues in the past, but I'm curious if our filesystem will deal with this number of files an still stay maintainable.

I learned that any overall-operation (like movie, backup) may become arbitrary long and filesystems suffer from heavy file numbers over time.

I thought about merging multiple items into one file (maybe replacing the last directory of your suggestion), but the very different item sizes and text blocks arriving unsorted but still must be finally delivered in their correct order are hard to handle.


Comment on Re^4: Store a huge amount of data on disk
Re^5: Store a huge amount of data on disk
by BrowserUk (Pope) on Oct 19, 2011 at 14:51 UTC
    but I'm curious if our filesystem will deal with this number of files an still stay maintainable.

    I guess the obvious question is: which file system? And can't you use a different one?

    You should also seriously investigate the various NoSQL DBs around. I guess you were hoping for more feedback on those with your question. but none were forthcoming.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re^5: Store a huge amount of data on disk
by zentara (Archbishop) on Oct 19, 2011 at 16:34 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://932288]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (4)
As of 2014-10-02 03:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (45 votes), past polls