in reply to
Re^4: Design flat files database
in thread Design flat files database
Thanks a lot! This forum rocks :)
Well, you're pretty much guaranteed to get your money's worth :-)
I like flat files because I can use all my favorite command line tools to manipulate them, without interacting with a database. And I have a lot of insight and control over what is involved in accessing them. As soon as you start aggregating the files because they may be "grabbed together", then some of the advantages of flat files start getting outweighed by the additional structure needed to aggregate them. Don't rule a real database out, they are good at such things.
Opening a file is not much different from searching a directory... you have to get the contents, and for flat files, that means going out to disk. If you keep related messages in the same file, you may be able to save disk accesses by getting more than one message with one read. But now you've made maintaining the collection of messages more difficult, and searching more complex, because you have different messages in the same file. A database may well be able to do this better.
As with directory structure, try to hide the implementation details. Start with single files, aggregating common files under a single directory. If the performance is good enough, this is likely to be a lot simpler than storing multiple messages in a single file. Above all, keep the user interface simple. They don't want to know about implementation details. And don't dismiss a database out of hand. It may make your life a lot easier, and perform at least as well as a cleverly crafted flat-file implementation