Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

by TomDLux (Vicar)
on Jul 25, 2004 at 02:06 UTC ( #377212=note: print w/replies, xml ) Need Help??


in reply to Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

If you really want to do things by yourself ....

Get a honking big file, say 4GB, and a smaller file, say 4 MB.

To manipulate file 17623, seek location 17623 * 4 in the smaller file, and read in the 4 byte word you find there, call it <offset>. Seek location <offset> in the larger file, and read bytes until you reach the EOF marker. Alternately, you could store a 2 byte length in the index file.

If you need to enlarge file N, copy it to the end of the large file, where you can do whatever you want. You can keep an index of unused 'holes', and move files that fit into the hole instead of to the end of the file. When the file runs out of space, compact the file to a new file, generating a new index.

On the other hand, DB companies spend millions of dollars and devote hundreds of employees to maximize the efficiency of such operations. Can you really out-perform them?

--
TTTATCGGTCGTTATATAGATGTTTGCA

  • Comment on Re: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://377212]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2021-10-27 22:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My first memorable Perl project was:







    Results (94 votes). Check out past polls.

    Notices?