Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

trouble parsing log file...

by perl_geoff (Acolyte)
on Nov 20, 2006 at 19:27 UTC ( #585115=perlquestion: print w/ replies, xml ) Need Help??
perl_geoff has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I am writing a perl script which will parse a log file on my server and display either a red, yellow or green light depending on if it catches an error or warning string in a webpage. If it doesn't see any of those strings, it will display a green light. I'm having trouble with my if and elsif statements that handle this. Right now my script parses my log file, but it only displays the green light, even though my sample logfile says "server DOWN." Also, I'm not sure if I should put these statements inside a loop; I think I may have to eventually, because my real log files will have several lines and will have to be parsed for all different types of warnings and errors. I tried a while loop with this, and got an infinite loop. Here's what I have so far:
$logfile="log.txt"; $error="DOWN"; $warn="PROBLEM"; $redbutton="\<img src\=\'default_files/perlredblink2\.gif'>"; $greenbutton="\<img src\=\'default_files/perlgreenblink\.gif'>"; $yellowbutton="\<img src\=\'default_files/perlyellowblink\.gif'>"; open LOG, $logfile or die "Cannot open $logfile for read :$!"; @logarray=<LOG>; # dumps all of $logfile into @logarray if (@logarray eq $error) { print "<!--Content-type: text/html-->\n"; print "$redbutton"; } elsif (@logarray eq $warn) { print "<!--Content-type: text/html-->\n"; print "$yellowbutton"; } else { print "<!--Content-type: text/html-->\n"; print "$greenbutton"; }
Thanks for any help!

Comment on trouble parsing log file...
Download Code
Re: trouble parsing log file...
by inman (Curate) on Nov 20, 2006 at 19:48 UTC
    The line @logarray=<LOG>;   # dumps all of $logfile into @logarray is reading all of the lines into an array. The test @logarray eq $error is in scalar context. It is comparing the number of lines in the file to the text. This will bever succeed.

    Ignore the list building and just work through the file line by line and use a regular expression to test.

    Take a look at the following as an example.

    use strict; # Set the button to green initially my $button = "perlgreenblink"; # test the file line by line. # The line gets read into $_ # I am testing on the DATA segment to illustrate the point while (<DATA>){ # test with a regex and end the # while loop if there is a problem if (/DOWN/){ $button = "perlredblink2"; last; } if (/PROBLEM/){ $button = "perlyellowblink"; last; } } print "HTML for <img src=\"$button.gif\" />\n"; __DATA__ nothing here going smoothly Its all going DOWN no PROBLEM at all

      Actually dumping the file to an array and parsing it element(line) by line is more efficient than parsing the file line by line.

      foreach my $element (@logarray) .

      You may want to close that open logfile after reading it into the array

        Dumping a multi Gigabyte log file into an array is going to get ugly quickly. A combination of the Perl internals and the IO buffering on the computer should take care of this situation line by line.

        I thought I'd reply rather than --ing your post just because I disagreed.

        I cannot think of any meaning of the phrase "more efficient" which would render your statement correct.

        All the reading I've ever done on the matter says that parsing a file line by line is extremely efficient. What happens is as follows. The operating system reads a chunk of the file into memory; this is then broken up on newlines (or whatever the value of $/ is); then we iterate over each line until we run out and the process repeats. We can parse a file line by line as follows:

        while ( <FILE> )

        If we choose to stop reading the file at any point (perhaps we've found what we want) and call last, then we end up only reading the smallest part of the file as necessary. This means it's efficient time-wise, and because we're only holding one chunk of file in memory at a time, it's efficient memory-wise.

        Alternately, my reading has said that "dumping the file to an array" and parsing it line by line is very inefficient. This is the case whether we do this like this:

        my @logarray = <FILE>; foreach my $element (@logarray)

        or like this:

        foreach my $element (<FILE>)

        This is because the file system still gives Perl the file on a chunk by chunk basis, and Perl still splits it up on $/, but Perl has to do this for the whole file even if we're only going to look at the first 10 lines. Worse, Perl now has to store the entire file in memory, rather than just a chunk. So this is the least efficient way to handle a file in Perl.

        It is however very useful when we need random access to the whole file; for example when sorting it, or pulling out random quotes.

        I'd love to hear why, if you think I'm mistaken in my understanding in this matter.

      Ok, I tried this, but could only get it to display the green button...also, I'm not sure I understand the logic of setting the button to green first. When I run your script it seems to ignore everything but the last line. Also, I don't believe any of my log files are above about 5 mb or so.
        The logic is that you are testing the log file for a specific condition. You create a starting condition that assumes that everything has gone well and would result in a green button. The idea is that if you get to the end of the file without hitting one of your two tests then everything was OK.

        You read the file one line at a time looking for either DOWN or PROBLEM. When one of these tests work, you set the button response accordingly and use last to leave the while loop and do something with the outcome.

        You may only be reading 5Mb of files but this translates into a much larger use of memory. It also involves the computer reading the file line by line anyway as it puts it into memory. If your match is on line 20 of a 2000 line file, your script only needs to read 20 lines and you are done.

        If you continue to have trouble, post your code in the replies.

Re: trouble parsing log file...
by liverpole (Monsignor) on Nov 20, 2006 at 19:52 UTC
    Hi perl_geoff,

    You are trying to compare a list against a scalar in this line:

    if (@logarray eq $error) {

    What I suspect you want to do instead is to iterate through the list, and break out of the loop when you find the condition you're looking for (note that you don't need to quote variable names to print them):

    chomp @logfile; my $got_button = 0; foreach my $line (@logarray) { if ($line eq $error) { $got_button = 1; print $redbutton; last; } elsif ($line eq $warn) { $got_button = 1; print $yellowbutton; last; } } if (not $got_button) { print $greenbutton; }

    The above will look through each line read in from the file, and print $redbutton or $yellowbutton if the appropriate string is matched.

    I also did a chomp @logfile to remove the newline from each logfile line, so that the match doesn't have to explicitly declare it.

    Also, the variable got_button was used so that, when you've gone through all the lines from the file, if the match wasn't found, you know to display the $greenbutton.

    Hope that helps!


    s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/
      This looks good logically, but I can only get it to display green again...my logfile specifically says "server is DOWN." Oh well, I will keep hackin around... :)
      Well, if you're looking for a pattern instead of an exact string, you can look into regular expressions.

      For example, to match the string "server is DOWN" anywhere in the line:

      if ($line =~ /server is DOWN/) { $got_button = 1; print $redbutton; last; }

      ... and you can ignore case (eg. match "Server IS down" and "SERVER is Down" both) by adding i at the end:

      if ($line =~ /server is DOWN/i) { $got_button = 1; print $redbutton; last; }

      But definitely look into regular expressions, which will open up a whole world for you in terms of pattern-matching power.


      s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/
        Thanks, I am about halfway through ch.8 'matching with regular expressions' in my learning perl book. Ok, I tried the following:
        $logfile="log.txt"; $error=(/DOWN/); $warn=(/PROBLEM/); $redbutton="\<img src\=\'default_files/perlredblink2\.gif'>"; $greenbutton="\<img src\=\'default_files/perlgreenblink\.gif'>"; $yellowbutton="\<img src\=\'default_files/perlyellowblink\.gif'>"; open LOG, $logfile or die "Cannot open $logfile for read :$!"; @logarray=<LOG>; # dumps all of $logfile into @logarray chomp @logfile; my $got_button = 0; foreach my $line (@logarray) { if ($line eq $error) { $got_button = 1; print $redbutton; last; } elsif ($line eq $warn) { $got_button = 1; print $yellowbutton; last; } } if (not $got_button) { print $greenbutton; }
        But it still only displays the green button!
        EDIT: Success! I got it now, thanks for your help everyone!
        use warnings; $logfile="log.txt"; $error="DOWN"; $warn="PROBLEM"; $redbutton="\<img src\=\'default_files/perlredblink2\.gif'>"; $greenbutton="\<img src\=\'default_files/perlgreenblink\.gif'>"; $yellowbutton="\<img src\=\'default_files/perlyellowblink\.gif'>"; open LOG, $logfile or die "Cannot open $logfile for read :$!"; my $button = $greenbutton; while (<LOG>) { if ($_ =~ /$error/i) { $button = $redbutton; print "<!--Content-type: text/html-->\n\n"; print "$redbutton"; } if ($_ =~ /$warn/i) { $button = $yellowbutton; print "<!--Content-type: text/html-->\n\n"; print "$yellowbutton"; } } close LOG;
Re: trouble parsing log file...
by madbombX (Hermit) on Nov 20, 2006 at 19:58 UTC
    You can't wrap a while loop around that to just have it continue to read the logfile either. Have a look at File::Tail for continuous logfile reading. Something like the following should work for you:
    use File::Tail; tie *LOG, 'File::Tail', (name => $logfile, tail => -1) or die("log open error: $!"); while (my $line = <LOG>) { if ($line eq $error) { print $redbutton; } # etc } close (LOG) or die("log close error: $!");
      Hi, I am new to using modules, can you give me more information on how to install/get File::Tail? Thanks!
        A Guide to Installing Modules

        One other small point that others seem to have missed....
        You probably don't want to break out of your while loop if you find a "PROBLEM". That is because there may be a "DOWN" further on in the logfile which you would then miss.

        ie. you should not have a last; in the if ($line =~ /$warn/) condition (or however you end up writing it).

        Cheers,
        Darren :)

Re: trouble parsing log file...
by web_developer (Initiate) on Jul 05, 2009 at 03:13 UTC
    OK, in light if this discussion, I have a similar problem.
    I need help, I have a script that i am trying to do the following:
    file1= list of unique ID numbers
    file2= list of unique html code with the unique ID numbers within the code per line(about 16k-bytes each seperated by carage.
    Both are standard text files.
    ===========
    #!/usr/bin/perl
    #read each line in test1.txt into data_file array
    $data_file="test1.txt";
    open(DATA, $data_file) || die("Could not open file!");
    @data_file=<DATA>;

    #read each line in code.txt into a names_file array
    $names_file="code.txt"
    open(NAMES, $names_file) || die("Could not open file!");
    @names_data=<NAMES>;

    #create loop that reads each ID in code.txt (NAMES array), searches for each in array elements for #test1.txt (DATA array), redirects a new (NAMES).html for each element
    foreach ( $NAMES )
    {
    chomp($NAMES);
    ($NAMES=$DATA<0> > +("$NAMES<0>.html"));
    }

    close NAMES;
    close DATA;

    I am new to perl but this is absolutely riddled with errors and I have written this according to examples of similar scripts.

    Node content restored by GrandFather

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://585115]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (11)
As of 2014-08-20 16:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (119 votes), past polls