Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Tail rotating logfiles with tie

by Brutha (Friar)
on Sep 30, 2004 at 09:44 UTC ( #395284=perlquestion: print w/replies, xml ) Need Help??

Brutha has asked for the wisdom of the Perl Monks concerning the following question:

Do You have long running applications, which write logfiles and do rotate them when they reach a maximum size? Do you need to 'tail -f' them?

As I could not find a solution, I try to solve this with a module using tie. I use ActiveState on Windows, the latter gives me some additional problems with file handling.

The following is more a proof of concept than a solution, even if it seems to work for me. The idea is to use normal file io-functions like open, readline or seek etc and tie this to a module, which transparently switches the files after rotation. The file must not stay open, otherwise the rotation of the locked file fails.

My logfile is written by an aplication with the great Log::Agent module combined with Log::Agent::Rotate. The important part of my reader is the standard solution for perl tail:

my $logfile = "logfile"; open( LOG, "<$logfile" ) or die "$logfile: $!"; while (!$end_condition) { while ( my $line = <LOG> ) { print $line; } sleep(1); # do something usefull seek( LOG, 0, 1 ); } close(LOG);
This way, it does not allow the logfile to rotate, as it is open. My solution inserts just a tie before the open-statement:
tie *LOG, "Tie::Tailrotate", "$logfile"; open ...
The module enabling the rotation is as follows (only main parts, see below for open questions)

package Tie::Tailrotate; use strict; # ;-) learned that use Carp; sub TIEHANDLE { my $class = shift; my $fname = shift; # need $fname to open file and initialize filehandle if ( $fname =~ /[>+|]/) { croak "Only read support for $fname\n"; } open my $self, $fname or croak "Open $fname failed:$!"; # $self is filehandle, # but created is $$$self, @$$self and %$$self # see Camel Book: 14.4. Tying Filehandles $$self->{filename} = $fname; $$self->{curpos} = 0; return bless $self, $class; # $self is a glob ref } sub OPEN { # needs to be covered my $self = shift; my $fname = shift; $self->CLOSE; if ( $fname =~ /[>+|]/) { croak "Only read support for $fname\n"; } open $self, $fname or croak "Open $fname failed:$!"; $$self->{curpos} = 0; $$self->{filename} = $fname; close($self); return 1; } sub READLINE { my $self = shift; my $line; my $size = (stat $$self->{filename} )[7]; # if the last position I read from is bigger than # actual file size, the file rotated. Continue # reading from last file, last position. my $use_active_file = ($$self->{curpos} <= $size ); if ($use_active_file) { # continue with active logfile open $self, $$self->{filename} or croak "Open $$self->{filename} failed:$!"; seek ($self, $$self->{curpos}, 0 ); $line = <$self>; # undef if eof $$self->{curpos} = tell($self) if ($line); } else { # File rotated, Log::Agent naming convention my $rot_fname = "$$self->{filename}.0"; $size = (stat $rot_fname )[7]; open $self, $rot_fname or croak "Open $rot_fname failed:$!"; seek ($self, $$self->{curpos}, 0 ); $line = <$self>; # undef if eof if ($line) { $$self->{curpos} = tell($self); } else { # reset position, to continue with active file $$self->{curpos} = 0; } } close($self); # unlock file return $line; }
I left out the seek, close etc. subs, the above should illustrate my concept. Remember, this is a first hit and needs lot of fine-tuning (or throw it away?) Your comments please.

And it has problems:

  • there is still achance, that the reader blocks the file, or hit a locked file (remove the sleep from the reader to see it).
  • the tie staement needs the filename to open the file and initialize $self with the filehandle (could/should be created manually)
  • better checks for readonly opening of files
  • implement a better seek for positioning
  • implement a better close
  • implement initial start position, e.g. start with last 10 lines instead of last 1000s from the beginning of a big logfile.
  • more general solution

And it came to pass that in time the Great God Om spake unto Brutha, the Chosen One: "Psst!"
(Terry Pratchett, Small Gods)

Replies are listed 'Best First'.
Re: Tail rotating logfiles with tie
by Roger (Parson) on Sep 30, 2004 at 11:36 UTC
    Have you thought about implementing semaphores to synchronize between your readers and writers?

      Semaphores are a good idea for different scenarios. My main application is a server performing downloads and compilations on demand. The reader is just a GUI added afterwards for people who prefer coloured pictures and 2-buttons over 100 keys. Both scripts are completly independent of one each other. I did not want to alter the source of the files.

      Nevertheless, it is good to have red coloured error messages.

      And it came to pass that in time the Great God Om spake unto Brutha, the Chosen One: "Psst!"
      (Terry Pratchett, Small Gods)

Re: Tail rotating logfiles with tie
by PotPieMan (Hermit) on Sep 30, 2004 at 15:21 UTC

    You might have found it in your research, but I've had luck with File::Tail. It lets you tie a filehandle to a rotating log file, automatically reopening the file if it detects that the file was truncated.

    CPAN Testers says it fails on Windows, but you might be able to look at it for ideas or submit patches.

    --Daniel

      Thank you for the hint, but I already looked at File::Tail. Tests fail under Windows, but to try minimum functinality this works.

      It also keeps the logfile open, which does not allow the logging program to rotate the file:

      Can't rename logfile to logfile.0: Permission denied

      And it came to pass that in time the Great God Om spake unto Brutha, the Chosen One: "Psst!"
      (Terry Pratchett, Small Gods)

Re: Tail rotating logfiles with tie
by aquarium (Curate) on Sep 30, 2004 at 13:47 UTC
    some of the unix system logrotate implementations i've seen have the rotation part scripted...hence you can plug your tail -f switching right into those.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://395284]
Approved by gellyfish
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (2)
As of 2021-05-09 03:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Perl 7 will be out ...





    Results (100 votes). Check out past polls.

    Notices?