Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

log file analysis

by vineet2004
on Dec 22, 2006 at 05:33 UTC ( #591257=perlquestion: print w/replies, xml ) Need Help??
vineet2004 has asked for the wisdom of the Perl Monks concerning the following question:

hi monks, i have a query. i wish to access a log file while it is being generated. what i really intend is to process the text in the log file. i am using AWK, but i can process only after the log is created. how can i process it while it is getting generated?

Replies are listed 'Best First'.
Re: log file analysis
by davidrw (Prior) on Dec 22, 2006 at 05:37 UTC

      I don't know the platform. On something un*x-like, the following should work (I assume a single logfile, without true rotation {:== writes modulo maxsize} but even that could be handled)

    • you read the file doing whatever filtering, and record size read
    • you seek to the last position recorded and filter from there
    • is there any reason why a simple scheme like this could not work? AND of course you should try what has been previously recommended

      hth --stephan
      thanks one more query coming up soon
Re: log file analysis
by bingos (Vicar) on Dec 22, 2006 at 08:05 UTC

      FYI, I've been using File::Tail for quite a while and it seems to handle file rotations also. I'm going to try out the POE version though. POE is very cool.


      I am using POE::Wheel::FollowTail and it is a great module. Handles log file rotation and truncation for you. The event driven system is ideal for loading various forwarders depending on what is seen in the logfile. It took me about a day to hammer together a state-full file follower that parses the file, forwards alerts and generates average stats from performance metrics in the log. POE rocks.


      Pereant, qui ante nos nostra dixerunt!
Re: log file analysis
by lovekumar (Initiate) on Dec 22, 2006 at 13:38 UTC
    You can try open a file in NON BLOCK MODE like below:-
    sysopen(RFD, "/usr/local/apache/logs/access_log", O_NONBLOCK); while(1){ my $buffer = <RFD>; # Do some thing with the line }
    I use this to check my access logs.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://591257]
Approved by prasadbabu
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (4)
As of 2019-02-17 11:22 GMT
Find Nodes?
    Voting Booth?
    I use postfix dereferencing ...

    Results (96 votes). Check out past polls.