Tail a rapidly changing file

[rah]

I am trying to extract data from a very rapidly changing log file. I would like to sample this file every 5 minutes and get the most recent 5 minutes worth of data. I don't really want to write a tail program myself. I am a firm believer of not re-inventing the wheel.

I had hoped to used File::Tail, or better yet File::Tail::App to perform the tail so I could focus on the parsing and reporting. I really like the lastrun capability of File::Tail::App and thought I could exploit that to help control the data sampling. However, I think that my data is changing too rapdily for this. Not long after I start the script, the lastrun file becomes zero length and the tail behaves like a unix tail -f. I think that because the existence of an EOF is so fleeting, that from the program's perspective there isn't one and so it continues to read.

What I'm wondering is if there is a way for me to tell File::Tail:App or the underlying File::Tail where I want it to stop. I could easily tell it the file size in bytes and therefore a place where it could end. But as far as I can tell there is no provision for a second seek value to denote the end of the data.

Am I just trying to ask these tail modules to do something they can't? Do I need to write my own tail functionality using seek or sysseek?. My test code at this point is nothing more that the example provided with the File::Tail::App module documentation but used on one of my log files. I haven't even started on the parsing logic.

#!/usr/bin/perl
use strict;
use warnings;
use Unix::PID '/var/run/tail.pid';
use File::Tail::App qw(tail_app);

   tail_app({
      'new'          => ['my.log'],
      'line_handler' => \&_wag_tail,
      'lastrun_file' => 'x.lastrun',
   });

   sub _wag_tail {
      my($line) = @_;
     print "$line \n";
   }

[sflitman]

I would just write my own code, it's pretty easy:

#!/usr/bin/perl
use strict;
use warnings;
use Fcntl qw/:seek/;

my $file='my.log';

# position to end
open(FILE,$file) || die "$file: $!";
seek(FILE,0,SEEK_END) || die "Seek: $!";
my $pos=tell(FILE);
close FILE;

while(1) {
   sleep(5*60);   # every 5 minutes
   open(FILE,$file) || die "$file: $!";
   seek(FILE,$pos,SEEK_SET) || die "Seek: $!";
   while (<FILE>) {
      #  do something with $_ lines
   }
   $pos=tell(FILE);   # update
   close FILE;
}

Send it the kill signal when you're done with it, or you could get fancy by adding code to abort when it sees something or other.

HTH,
SSF

[matze77]

Hello!
I dont know what use Fcntl qw/:seek/; accomplishes if someone got the time maybe he/she could explain?
Thanks
MH

[BrowserUk]

I dont know what use Fcntl qw/:seek/; accomplishes ...

It imports the constants SEEK_SET SEEK_CUR SEEK_END.

[Khen1950fx]

Try File::ReadBackwards and read grinder's Performing a tail(1) in Perl (reading the last N lines of a file).

[rah]

I looked at both earlier, but I don't think either will apply. I have the problem Grinder describes about not being able to stop File::Tail. But I don't think any of the suggestions in the linked node will work.

I don't think ReadBackwards will work because I don't know how many lines I need. It will be some very large number most times and highly variable at all times.