After a good example of efficient record handling...

[NeilF]

My current forum uses flat text files and reads/writes the entire file when adding a new message.

I'd like to look into using fixed length files for indexes etc for faster and more efficient data handling.

I've searched, googled and even downloaded example forums, but cannot see examples.

Can anyone point me to some example of good record handling - eg using seek etc to only update the records required. If its code specifically for a forum all the better!!!

Edited 1 Jan 2006, by footpad: Added basic formatting tags.

[chromatic]

Are you sure you want to use fixed-length record sizes for forum posts? Do you have a small maximum length for the body of the post? If not, all of your records will be the size of the largest possible record -- and if that's not small, you'll have to do a lot of IO to read each message from the disk, pull it through the cache, and get it where perl can handle it. If you really think that will improve your speed (and I don't, even without benchmarking it), you need some sort of unique ID per message which you can use as an offset into the file. Multiply that by the size of a record and use seek and sysread on a filehandle to extract just the record you want.

If that doesn't seem fun to you (and it's not), you could use a unique record separator between records and let Tie::File take care of reading and writing.

If this were my project, I would use DBD::SQLite and avoid all of the tedious mucking about with O(n) access and let indexed columns get that down to O(log n) or better, while not having to process all of the data myself. Please consider that instead.

[holli]

Taking your performance/io problems in Perl always reads in 4K chunks and writes in 1K chunks... Loads of IO! into account, i think you could use a flat file for storing post titles and a reference to a file that contains the actual post.

[NeilF]

I agree, but I can see lots of problems/unkowns with this and would like to see and article or even better a working example...

[TedPride]

See pack and unpack for record handling; sysopen, sysseek, sysread, syswrite for retrieving or writing records; and flock for keeping the files from being modified during writes. As mentioned above, post contents should be in a separate file, with the post record containing just start and end offsets. This keeps your records neat and fixed-length. If your users are allowed to do edits, you should allocate a few extra characters for post content so small increases don't require moving it to a new spot in the file. After the time limit for edits is reached, your clean-up utility can rewrite the content to exact size.

It would, however, be simpler to just do this with mySQL. Certain things like indexes can get quite complicated to program.