Reading log files that get overwritten

[Zapawork]

So here is my problem,
I have a Nokia internet appliance running a modified version of bsd. It is running checkpoint firewall 1, this is just for background knowledge, so none of the solutions can be modify the software.

It is creating a log file that grows to a specified size and then starts writing back at 0 again.

I want to be able to continously read this log file, determine when it has reset it's pointer and then read from the top again. I need to then send this information off via syslog to another server.

Any ideas?

Dave -- Saving the world one node at a time

[zengargoyle]

File-Tail will likely do what you want.

it will notice that there hasn't been any updates to the end of the file and will re-open and start at the beginning

in this way it handles such things as:

• file being truncated. (cat /dev/null > logfile)
• file being moved. (mv logfile logfile.old; touch logfile)
try it, you'll like it.

[Zapawork]

This sounds like a winner,
Thanks for all your help!

Dave -- Saving the world one node at a time

[slugmax]

Is this a text file, or a binary one (e.g. $FWDIR/log/fw.log is a binary file). Is it being written by the system or by FireWall-1, or by a script you wrote? I only ask because FireWall-1 has a command-line interface to its binary log format that may do what you want, and is scriptable.

If possible, the best way to deal with files like this is to not even bother reading them - but by sending the data from the generating process in realtime over a network socket, much as remote syslog does. That way, you don't have to worry about the file contents.

If the above is not an option, you could try File::Tail. Nokia's "unsupported" perl package installs in /opt, which *is* mounted read/write (unlike the root partition) so you should be able to make it work by installing it in one of the @INC directories, which are all under /opt/perl/lib/perl5.

Doug

[Zapawork]

Hi Doug,
As I noted in a later reply, the command line utility is the source of my problem. It reads to EOF and then stops. So my solution is to read fw.log directly and then interpret the data on the remote end. So I am going to try the file::tail solution that was brought up earlier.

Thanks for the help!
Dave -- Saving the world one node at a time

[Anonymous Monk]

I just had a perl epiphany!
Be aggressive! Do this:

Assuming that the data to be written will buffer and then
be written to the file in a large chunk --> lock the file so that the FWall cant write to it.

o Wipe the file clean
o Release the lock
o Lock again
o Read file's contents (which have been written when you released the lock )
o Send the contents to syslog
o Wipe file clean
o Unlock again and repeat the whole operation

I think this should work very well.
-bl0rf

[waswas-fng]

Bad, FW1's log write queue has a finite length and blocking it may lead to blocking on the packet inspector on a busy firewall.

-Waswas

[Zapawork]

Waswas is right ... The point of this exercise is to have a secondary source for log data incase the management server where to fail.

Dave -- Saving the world one node at a time

[steves]

I'd have to see your read loop to see how you're handling end of file, but I'd assume you could somehow stat the file in your loop and see if the size changed since the last read.

[waswas-fng]

A non perl solution is this:

 $FWDIR/bin/fw log -f 2>>/var/adm/fw-log.log | /bin/logger -p local5.i
+nfo > /dev/null 2>&1 &

and set the syslog conf to point to the syslog server.

-Waswas

[Zapawork]

Hi waswas,
That was the first thing I tried, however what I noticed was that the utility stops after a period of time since it is reading from the same file and does not repeat over it to determine the next block. Thus my problem... pardon me if that is nonsensical.. it's late.

Dave -- Saving the world one node at a time

[waswas-fng]

What Nokia model are you using? I have never seen that behavior before.

-Waswas

[kschwab]

Check and see if the inode number changes ( ls -li or stat() ) when the file is truncated.

If so, you can just make a hard link, and then note when the inode numbers don't match anymore.

This also has the advantage of preserving the old data.

If, on the other hand, they are calling truncate(), you'll need to resort to watching file sizes.

[hardburn]

Are you trying to write to a file that you are reading at the same time? That's just asking for problems. Imagine if you were trying to read a document while the author stands over your shoulder and scribbles notes on it while you were trying to read it.

[Zapawork]

No I'm trying to read a file and then send the contents of it to a remote machine.

Dave -- Saving the world one node at a time