Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re^5: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

by davido (Cardinal)
on Jul 24, 2004 at 06:15 UTC ( #377092=note: print w/replies, xml ) Need Help??


in reply to Re^4: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)
in thread Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

If speed meant NOTHING you'd do it by hand.

The point is that inserting into a flat file means rewriting the entire file each time, unless you store updates and do them in groups. Imagine, if your 2mb flat file grows by 4 bytes each iteration, you're moving around 2mb of data each time you try to add 4 bytes. That's a 2,000,000 bytes of data reading and rewriting for each 4 byte insertion. Speed has got to mean something.

I think your solution may be a good one. If building up the dataset is a one-time deal, do it in a database, and then transfer the completed product to a flat file where it can be read quickly.


Dave

  • Comment on Re^5: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

Replies are listed 'Best First'.
Re^6: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)
by rjahrman (Scribe) on Jul 24, 2004 at 07:19 UTC
    Sorry if I came off as a bit ignorant. :)

    I agree that moving that data around like that is not an option. I just meant that if building this file take twice as long, but yields a file 10% smaller, it's worth it in this particular case. The file will only be built occasionally (like every other month).

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://377092]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (2)
As of 2021-10-27 23:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My first memorable Perl project was:







    Results (95 votes). Check out past polls.

    Notices?