Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Large chunks of text - database or filesystem?

by cyocum (Curate)
on Mar 19, 2005 at 10:09 UTC ( #440901=note: print w/replies, xml ) Need Help??


in reply to Large chunks of text - database or filesystem?

I would use a database for your problems. The reason? Ease of transfering the information from one machine to another. All the information would be stored in a few files and can be transfered to a new database much more easily. Hard drive space is releatively cheap these days and having millions of files all over the place is ineffiecant in my opinion (as always, I could be terribly wrong).

For effecient searching, I would use Plucene. Internally, MyISAM tables do have the ability to compress but only for read-only tables. I would run Plucene as a cron or scheduled task or have it triggered by an insertion. Do not worry Plucene (at least as far as I know for Lucene from which Plucene came) should store partial indexes in memory before flushing to disk so it should be pretty quick about additions to the index.

About the storage requirements for text, check the MySQL documentation here. That should let you know how much disk space you are wasting with each post.

I hope this helps.

  • Comment on Re: Large chunks of text - database or filesystem?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://440901]
help
Chatterbox?
[choroba]: their = his or her
[Discipulus]: anyway once I proposed an automatic font decrease for post below -1, stepping every 5 votes: i'm still thinks it's a good idea but dunno if feasible
[shmem]: ...emulated dual in english
[shmem]: Discipulus: has that been in the CB? or in a PM node proper? for the latter case, can you provide the ID?
[Discipulus]: you seriously can use their in this way in eng?
[Corion]: Meh. $effin_bad_system has an interface breakdown and then loads events in parallel with events overtaking one another instead of being processed sequentially
[shmem]: Discipulus: dunno, but we do all the time ^^
[choroba]: Discipulus I was taught so by a Londoner
[shmem]: Corion: very clear case of missing sequence number
[Corion]: shmem: Yeah. I guess they have a sequence number but distribute the events across threads or machines or whatever.

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (11)
As of 2017-05-23 08:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?