We don't bite newbies here... much | |
PerlMonks |
Re^2: Using binary search to get the last 15 minutes of httpd access logby Old_Gray_Bear (Bishop) |
on Aug 04, 2012 at 15:42 UTC ( [id://985440]=note: print w/replies, xml ) | Need Help?? |
I had to solve the same problem (for Apache logs, too) a few years back. Brute force is fine for a small log, the logs I was parsing were growing at a gigabyte+ per minute. (We rolled logs every 100 GB or 30 minutes, which ever came first.)
Pseudo code: This gimmick ran (most of the time) in under 500 milli-seconds, and gave us enough information. The Perl implementation was fast enough (most times) that we never got around to implementing it in C. You can run into problems with slow growing logs (what happens if there is only one line in the file?), and mumungous lines (again, only one line in the file and its 55MB long!). We got around it by fiat -- if something goes sour, quit; and retry again in 30 seconds. (Yahoo, Instant Messenger, three to four terabytes of logs per day....)
---- OGB
In Section
Seekers of Perl Wisdom
|
|