Trying to identify unique lines in a log file

[aditya1977]

I'm writing a program that will parse a log file and write data to a MySQL database based on the content of the log file.

The script will run via cron and I want to make sure I skip lines that have been parsed before.

One idea I had was to base 64 encode the line and make this a unique key in the database. Then the next time the script runs, if there is a matching key, skip the line.

What I don't like about this is that the encoded line is quite a long string, which would make viewing the database difficult to read. Is there a way to encode a long string to a relatively short string string?

Or perhaps there is a better way to do this?

[choroba]

For shorter strings, see Digest::MD5 and similar. Also, if the log grows on one end only, it should be enough to just remember the last line.

[aditya1977]

Thanks! You made me realise that there's no reason not to use hashing rather than encoding.

And MD5 hashes are significantly shorter.

Timestamp entries are present, but not unique as some events take place milliseconds apart.

[Theodore]

Is there any timestamp in the log? If not, can you add one? Or even, Why not cycle the log first (copy and truncate) and then parse the "static" recycled log?

[GotToBTru]

To reprocess the log file each time seems a terrible waste of time and resources, not to mention storing your ever-growing index of lines you have already logged! Use seek() and tell(). You need only store a bookmark and you will know where to start from next time. This will need to be coordinated with whatever process truncates your log file, of course.