Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

After a good example of efficient record handling...

by NeilF (Sexton)
on Jan 01, 2006 at 21:18 UTC ( #520286=perlquestion: print w/replies, xml ) Need Help??

NeilF has asked for the wisdom of the Perl Monks concerning the following question:

My current forum uses flat text files and reads/writes the entire file when adding a new message.

I'd like to look into using fixed length files for indexes etc for faster and more efficient data handling.

I've searched, googled and even downloaded example forums, but cannot see examples.

Can anyone point me to some example of good record handling - eg using seek etc to only update the records required. If its code specifically for a forum all the better!!!

Edited 1 Jan 2006, by footpad: Added basic formatting tags.

  • Comment on After a good example of efficient record handling...

Replies are listed 'Best First'.
Re: After a good example of efficient record handling...
by chromatic (Archbishop) on Jan 01, 2006 at 21:49 UTC

    Are you sure you want to use fixed-length record sizes for forum posts? Do you have a small maximum length for the body of the post? If not, all of your records will be the size of the largest possible record -- and if that's not small, you'll have to do a lot of IO to read each message from the disk, pull it through the cache, and get it where perl can handle it. If you really think that will improve your speed (and I don't, even without benchmarking it), you need some sort of unique ID per message which you can use as an offset into the file. Multiply that by the size of a record and use seek and sysread on a filehandle to extract just the record you want.

    If that doesn't seem fun to you (and it's not), you could use a unique record separator between records and let Tie::File take care of reading and writing.

    If this were my project, I would use DBD::SQLite and avoid all of the tedious mucking about with O(n) access and let indexed columns get that down to O(log n) or better, while not having to process all of the data myself. Please consider that instead.

Re: After a good example of efficient record handling...
by holli (Abbot) on Jan 01, 2006 at 23:37 UTC
      I agree, but I can see lots of problems/unkowns with this and would like to see and article or even better a working example...
Re: After a good example of efficient record handling...
by TedPride (Priest) on Jan 02, 2006 at 09:23 UTC
    See pack and unpack for record handling; sysopen, sysseek, sysread, syswrite for retrieving or writing records; and flock for keeping the files from being modified during writes. As mentioned above, post contents should be in a separate file, with the post record containing just start and end offsets. This keeps your records neat and fixed-length. If your users are allowed to do edits, you should allocate a few extra characters for post content so small increases don't require moving it to a new spot in the file. After the time limit for edits is reached, your clean-up utility can rewrite the content to exact size.

    It would, however, be simpler to just do this with mySQL. Certain things like indexes can get quite complicated to program.

A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://520286]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (1)
As of 2022-05-18 04:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (68 votes). Check out past polls.

    Notices?