Re: Using binary search to get the last 15 minutes of httpd access log

[flexvault]

mhearse,

If you are doing this once, then reading the whole file is fine, but if every 'nn' minutes you want to run the script then saving the location of the end-of-file would be a good solution. This technique is similar to 'pop-before-smtp' technique. (Example, not tested )

  my $File_Loc = 0; my $SaveLocFile = "./FileLoc";
  if ( -e $SaveLocFile )
  {   open( my $IN, "<", $SaveLocFile ) or die "File Bad $!";
      $File_Loc = < $IN >; chomp ( $File_Loc );
      close $IN;
  }

  my $sz = ( -s $SaveLocFile );
  if ($sz > 0)
  {  open( my $log, "<", $log_file");          ## This should be log f
+ile
     if ($sz < $File_Loc ) { $File_Loc = 0; }  ## New log file
     seek $log, $File_Loc, 0;                  ## Move to old end of f
+ile
     while ( < $log > )
     {   chomp;
         my $line = $_;
         # do work 
      }

      $File_Loc = tell $log;      ## Save current end-of-file location
      open( my $OUT,">", $SaveLocFile );
      print $OUT "$File_Loc\n";
      close $OUT;
      close $log;
    }

Since you will be running on a log server, you should use the same locking mechanism as the log daemon. There are some race conditions in the code, but if you save where you ended, you shouldn't miss anything. I originally wrote this in 1997, so I tried to update the technique. Looking at the original code, it isn't the way I would write it today, but it has worked on some 40 - 50 mail servers since, and I never had to rewrite it (probably should now).

Hopefully you see the idea and can adapt this to your situation.

Good Luck