Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

How to read from rotated log file?

by Wen (Initiate)
on Jan 29, 2012 at 21:05 UTC ( #950657=perlquestion: print w/replies, xml ) Need Help??
Wen has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks, I am a Perl newbie and hope you can point me in the right direction. I am writing script, which will look at a certain records in system log files. The problem is that it may need to check the records going up to few days back. By the time of running the check, log files may have already been rotated and archived as bz2 in the same /var/log directory.

What is the most efficient way to "go back in time" and search in these rotated files? Since logs are rotated by size, not date, the script does not know through how many files back it has to look, until it looks inside the file.

Thank you for advice

Replies are listed 'Best First'.
Re: How to read from rotated log file?
by JavaFan (Canon) on Jan 29, 2012 at 21:16 UTC
    I assume your system numbers log files, and that compressed log files end in .gz.

    Let's assume your base file is called "my-app". Then it's time to use magic open! Untested:

    my @files = </var/log/my-app*>; /\.gz/ and $_ = "zcat $_ |" for @files; local @ARGV = @files; while (<>) { ... do something with record ... }
    This will read uncompressed files in the usual way, and read from a pipe (with zcat on the other end) for compressed files.
Re: How to read from rotated log file?
by johngg (Abbot) on Jan 29, 2012 at 23:47 UTC

    If you know the date you want to go back to then you should be able to convert that date into epoch seconds (see Time::Local) and compare with the time last modified (see stat) of your compressed log files to see which one to examine.

    If you are using 5.12 or later you should be able to use the built-in IO::Uncompress::Bunzip2 module to read the file. 5.10 or earlier and you will have to use the bunzip2 executable the way JavaFan has shown for gunzip.



      Thank you, JohnGG. It is a great idea to look at the file creation dates and I used the piece of code by JavaFan with slight modifications. Thanks for your help, guys!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://950657]
Approved by Corion
NodeReaper wonders through a maze of twisty little sayings, all alive

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2018-01-19 14:43 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (219 votes). Check out past polls.