Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Reading log files that get overwritten

by Zapawork (Beadle)
on Jan 02, 2003 at 18:22 UTC ( #223822=perlquestion: print w/replies, xml ) Need Help??
Zapawork has asked for the wisdom of the Perl Monks concerning the following question:

So here is my problem,
I have a Nokia internet appliance running a modified version of bsd. It is running checkpoint firewall 1, this is just for background knowledge, so none of the solutions can be modify the software.

It is creating a log file that grows to a specified size and then starts writing back at 0 again.

I want to be able to continously read this log file, determine when it has reset it's pointer and then read from the top again. I need to then send this information off via syslog to another server.

Any ideas?

Dave -- Saving the world one node at a time

Replies are listed 'Best First'.
Re: Reading log files that get overwritten
by zengargoyle (Deacon) on Jan 02, 2003 at 23:55 UTC

    File-Tail will likely do what you want.

    it will notice that there hasn't been any updates to the end of the file and will re-open and start at the beginning

    in this way it handles such things as:

    • file being truncated. (cat /dev/null > logfile)
    • file being moved. (mv logfile logfile.old; touch logfile)
    try it, you'll like it.

      This sounds like a winner,
      Thanks for all your help!

      Dave -- Saving the world one node at a time

Re: Reading log files that get overwritten
by slugmax (Hermit) on Jan 02, 2003 at 20:10 UTC
    Is this a text file, or a binary one (e.g. $FWDIR/log/fw.log is a binary file). Is it being written by the system or by FireWall-1, or by a script you wrote? I only ask because FireWall-1 has a command-line interface to its binary log format that may do what you want, and is scriptable.

    If possible, the best way to deal with files like this is to not even bother reading them - but by sending the data from the generating process in realtime over a network socket, much as remote syslog does. That way, you don't have to worry about the file contents.

    If the above is not an option, you could try File::Tail. Nokia's "unsupported" perl package installs in /opt, which *is* mounted read/write (unlike the root partition) so you should be able to make it work by installing it in one of the @INC directories, which are all under /opt/perl/lib/perl5.

      Hi Doug,
      As I noted in a later reply, the command line utility is the source of my problem. It reads to EOF and then stops. So my solution is to read fw.log directly and then interpret the data on the remote end. So I am going to try the file::tail solution that was brought up earlier.

      Thanks for the help!
      Dave -- Saving the world one node at a time

Re: Reading log files that get overwritten
by steves (Curate) on Jan 02, 2003 at 18:37 UTC

    I'd have to see your read loop to see how you're handling end of file, but I'd assume you could somehow stat the file in your loop and see if the size changed since the last read.

Re: Reading log files that get overwritten
by Anonymous Monk on Jan 02, 2003 at 19:05 UTC
    I just had a perl epiphany!
    Be aggressive! Do this:

    Assuming that the data to be written will buffer and then
    be written to the file in a large chunk --> lock the file so that the FWall cant write to it.

    o Wipe the file clean
    o Release the lock
    o Lock again
    o Read file's contents (which have been written when you released the lock )
    o Send the contents to syslog
    o Wipe file clean
    o Unlock again and repeat the whole operation

    I think this should work very well.

      Bad, FW1's log write queue has a finite length and blocking it may lead to blocking on the packet inspector on a busy firewall.

        Waswas is right ... The point of this exercise is to have a secondary source for log data incase the management server where to fail.

        Dave -- Saving the world one node at a time

Re: Reading log files that get overwritten
by kschwab (Priest) on Jan 02, 2003 at 22:05 UTC
    Check and see if the inode number changes ( ls -li or stat() ) when the file is truncated.

    If so, you can just make a hard link, and then note when the inode numbers don't match anymore.

    This also has the advantage of preserving the old data.

    If, on the other hand, they are calling truncate(), you'll need to resort to watching file sizes.

Re: Reading log files that get overwritten
by hardburn (Abbot) on Jan 02, 2003 at 18:28 UTC

    Are you trying to write to a file that you are reading at the same time? That's just asking for problems. Imagine if you were trying to read a document while the author stands over your shoulder and scribbles notes on it while you were trying to read it.

      No I'm trying to read a file and then send the contents of it to a remote machine.

      Dave -- Saving the world one node at a time

Re: Reading log files that get overwritten
by waswas-fng (Curate) on Jan 03, 2003 at 01:14 UTC
    A non perl solution is this:
    $FWDIR/bin/fw log -f 2>>/var/adm/fw-log.log | /bin/logger -p local5.i +nfo > /dev/null 2>&1 &
    and set the syslog conf to point to the syslog server.

      Hi waswas,
      That was the first thing I tried, however what I noticed was that the utility stops after a period of time since it is reading from the same file and does not repeat over it to determine the next block. Thus my problem... pardon me if that is nonsensical.. it's late.

      Dave -- Saving the world one node at a time

        What Nokia model are you using? I have never seen that behavior before.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://223822]
Approved by adrianh
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (9)
As of 2017-09-26 22:06 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (298 votes). Check out past polls.