Tail rotating logfiles with tie

[Brutha]

Do You have long running applications, which write logfiles and do rotate them when they reach a maximum size? Do you need to 'tail -f' them?

As I could not find a solution, I try to solve this with a module using tie. I use ActiveState on Windows, the latter gives me some additional problems with file handling.

The following is more a proof of concept than a solution, even if it seems to work for me. The idea is to use normal file io-functions like open, readline or seek etc and tie this to a module, which transparently switches the files after rotation. The file must not stay open, otherwise the rotation of the locked file fails.

My logfile is written by an aplication with the great Log::Agent module combined with Log::Agent::Rotate. The important part of my reader is the standard solution for perl tail:

my $logfile = "logfile";
open( LOG, "<$logfile" ) or die "$logfile: $!";

while (!$end_condition)
{
    while ( my $line = <LOG> ) {
        print $line;
    }
    sleep(1); # do something usefull 
    seek( LOG, 0, 1 );
}
close(LOG);

This way, it does not allow the logfile to rotate, as it is open. My solution inserts just a tie before the open-statement:

tie *LOG, "Tie::Tailrotate", "$logfile"; 
open ...

The module enabling the rotation is as follows (only main parts, see below for open questions)

package Tie::Tailrotate;
use strict;  # ;-) learned that
use Carp;

sub TIEHANDLE {
    my $class = shift;
    my $fname = shift; 
    # need $fname to open file and initialize filehandle
    if ( $fname =~ /[>+|]/) {
        croak "Only read support for $fname\n"; }
    open my $self, $fname
        or croak "Open $fname failed:$!";
    # $self is  filehandle, 
    # but created is $$$self, @$$self and %$$self
    # see Camel Book: 14.4. Tying Filehandles
    $$self->{filename} = $fname;
    $$self->{curpos}   = 0;    
    return bless $self, $class;    # $self is a glob ref
}

sub OPEN { # needs to be covered
    my $self  = shift;
    my $fname = shift;
    $self->CLOSE;
    if ( $fname =~ /[>+|]/) {
        croak "Only read support for $fname\n"; }
    open $self, $fname
        or croak "Open $fname failed:$!";
    $$self->{curpos}   = 0;    
    $$self->{filename} = $fname;
    close($self);
    return 1;
}

sub READLINE {
    my $self = shift;
    my $line;
    my $size  = (stat $$self->{filename} )[7];
    # if the last position I read from is bigger than
    # actual file size, the file rotated. Continue
    # reading from last file, last position.
    my $use_active_file = ($$self->{curpos} <= $size );
    if ($use_active_file)
    {    # continue with active logfile
        open $self, $$self->{filename}
            or croak "Open $$self->{filename} failed:$!";
        seek ($self, $$self->{curpos}, 0 );
        $line = <$self>;  # undef if eof
        $$self->{curpos} = tell($self)  if ($line);
    } 
    else {  # File rotated, Log::Agent naming convention
        my $rot_fname = "$$self->{filename}.0";
        $size = (stat $rot_fname )[7];
        open $self, $rot_fname
            or croak "Open $rot_fname failed:$!";
        seek ($self, $$self->{curpos}, 0 );
        $line = <$self>; # undef if eof
        if ($line) { $$self->{curpos} = tell($self); }
        else 
        {  # reset position, to continue with active file
            $$self->{curpos} = 0;
        }
    }
    close($self); # unlock file
    return $line;
}

I left out the seek, close etc. subs, the above should illustrate my concept. Remember, this is a first hit and needs lot of fine-tuning (or throw it away?) Your comments please.

And it has problems:

• there is still achance, that the reader blocks the file, or hit a locked file (remove the sleep from the reader to see it).
• the tie staement needs the filename to open the file and initialize $self with the filehandle (could/should be created manually)
• better checks for readonly opening of files
• implement a better seek for positioning
• implement a better close
• implement initial start position, e.g. start with last 10 lines instead of last 1000s from the beginning of a big logfile.
• more general solution

[Roger]

Have you thought about implementing semaphores to synchronize between your readers and writers?

[Brutha]

Semaphores are a good idea for different scenarios. My main application is a server performing downloads and compilations on demand. The reader is just a GUI added afterwards for people who prefer coloured pictures and 2-buttons over 100 keys. Both scripts are completly independent of one each other. I did not want to alter the source of the files.

Nevertheless, it is good to have red coloured error messages.

[PotPieMan]

You might have found it in your research, but I've had luck with File::Tail. It lets you tie a filehandle to a rotating log file, automatically reopening the file if it detects that the file was truncated.

CPAN Testers says it fails on Windows, but you might be able to look at it for ideas or submit patches.

[Brutha]

Thank you for the hint, but I already looked at File::Tail. Tests fail under Windows, but to try minimum functinality this works.

It also keeps the logfile open, which does not allow the logging program to rotate the file:

 Can't rename logfile to logfile.0: Permission denied

[aquarium]

some of the unix system logrotate implementations i've seen have the rotation part scripted...hence you can plug your tail -f switching right into those.