Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Using binary search to get the last 15 minutes of httpd access log

by flexvault (Parson)
on Aug 04, 2012 at 09:04 UTC ( #985408=note: print w/ replies, xml ) Need Help??


in reply to Using binary search to get the last 15 minutes of httpd access log

mhearse,

If you are doing this once, then reading the whole file is fine, but if every 'nn' minutes you want to run the script then saving the location of the end-of-file would be a good solution. This technique is similar to 'pop-before-smtp' technique. (Example, not tested )

my $File_Loc = 0; my $SaveLocFile = "./FileLoc"; if ( -e $SaveLocFile ) { open( my $IN, "<", $SaveLocFile ) or die "File Bad $!"; $File_Loc = < $IN >; chomp ( $File_Loc ); close $IN; } my $sz = ( -s $SaveLocFile ); if ($sz > 0) { open( my $log, "<", $log_file"); ## This should be log f +ile if ($sz < $File_Loc ) { $File_Loc = 0; } ## New log file seek $log, $File_Loc, 0; ## Move to old end of f +ile while ( < $log > ) { chomp; my $line = $_; # do work } $File_Loc = tell $log; ## Save current end-of-file location open( my $OUT,">", $SaveLocFile ); print $OUT "$File_Loc\n"; close $OUT; close $log; }

Since you will be running on a log server, you should use the same locking mechanism as the log daemon. There are some race conditions in the code, but if you save where you ended, you shouldn't miss anything. I originally wrote this in 1997, so I tried to update the technique. Looking at the original code, it isn't the way I would write it today, but it has worked on some 40 - 50 mail servers since, and I never had to rewrite it (probably should now).

Hopefully you see the idea and can adapt this to your situation.

Good Luck

"Well done is better than well said." - Benjamin Franklin


Comment on Re: Using binary search to get the last 15 minutes of httpd access log
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://985408]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (7)
As of 2014-09-16 11:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (12 votes), past polls