|Do you know where your variables are?|
Trying to identify unique lines in a log fileby aditya1977 (Novice)
|on Mar 05, 2015 at 10:21 UTC ( #1118867=perlquestion: print w/replies, xml )||Need Help??|
aditya1977 has asked for the wisdom of the Perl Monks concerning the following question:
I'm writing a program that will parse a log file and write data to a MySQL database based on the content of the log file.
The script will run via cron and I want to make sure I skip lines that have been parsed before.
One idea I had was to base 64 encode the line and make this a unique key in the database. Then the next time the script runs, if there is a matching key, skip the line.
What I don't like about this is that the encoded line is quite a long string, which would make viewing the database difficult to read. Is there a way to encode a long string to a relatively short string string?
Or perhaps there is a better way to do this?
Back to Seekers of Perl Wisdom