Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Search logs with crontab

by hmb104 (Sexton)
on Nov 18, 2013 at 16:09 UTC ( #1063138=perlquestion: print w/ replies, xml ) Need Help??
hmb104 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I'm trying to search logs every 2 hours for string, and if found then sendmail should kick in. I'm trying to find the best way to run my cron jobs every 2 hours without getting repetitive emails.

I decided just to search for the string, push it into my array, then change it's format so the next run for my script it doesn't catch it.

open SwitchLogs, $logFile or die "Could not open $logFile: $!"; while (<SwitchLogs>) { chomp; my $line = $_; if($line =~ m/PHY-4-EXCESSIVE_ERRORS/){ push(@err, "$line\n"); s/PHY-4-EXCESSIVE_ERRORS/PHY_4_EXCESSIVE-ERRORS/g; $sendemail = 1; } } close SwitchLogs;

I need help with the replace, how do I make it do the replace? Currently the script runs but still every 2 hours I keep getting the same email because the replace doesn't work. I know I'm missing something very basic.

Comment on Search logs with crontab
Download Code
Re: Search logs with crontab
by toolic (Chancellor) on Nov 18, 2013 at 16:17 UTC
    Your substitution operates on the contents of the $_ variable, but then you never do anything with that (potentially) modified variable. Did you not show a complete code example? I expected to see a print somewhere.

      That is my problem, I want to know how to replace the specified keyboard in that line I hold in my special variable?

      I was missing the file handler to write to the file and I got it resolved. But now my script works only on old logs. I want to do something with my current log file that are running 24/7.

      I want not to search from the top of the file again if I already did 2 hours a go, any ideas?

Re: Search logs with crontab
by taint (Chaplain) on Nov 18, 2013 at 16:22 UTC
    I might also add, that you could put your script into a sleep for 2hrs. time, before running again. Thereby eliminating the repetitive mail(s) cron sends.

    --Chris

    #!/usr/bin/perl -Tw
    use Perl::Always or die;
    my $perl_version = (5.12.5);
    print $perl_version;

      The 2 hours sleep will not help me here. The script will still run and catch the same old logs it did before and email me about them. How can I search only from that specific time and onward?

        Take a look at the seek function.

        If I understand you correctly. That was my point. Given that cron will alert you every time it fires off your job, with a message indicating the results. It seemed quite pointless to use cron to manage your script (for the most part). I thought it better, since you're already utilizing sendmail to deliver the outcome of your script. Why not fire your Perl script, and let it manage the schedule thru sleep. Maybe via a for loop, or something. You could capture the entire process via print, or simply redirect <, > the output/results to a file, and have sendmail send it to you, via attachement, or some such thing.

        Best wishes.

        --Chris

        #!/usr/bin/perl -Tw
        use Perl::Always or die;
        my $perl_version = (5.12.5);
        print $perl_version;

        The idea is, you save the size of the file that you've checked at one point of time, and then next time you skip this much of data before you start looking for your string. Then you don't need to modify anything in the file.

        The easiest way to do it is to let perl run forever, letting it sleep for 2 hours and then repeating the loop. Then you can keep the old file size simply in a variable. Beware, if you restart the script, you'll loose the offset.

        If you insist on running perl from crontab, then you'll have to save the size in some file and then on startup read that file to obtain the offset of data where searching should start.

        Note, there's still one catch here. Most likely the log file is rotated periodically, so your program should be smart enough to notice that log file was rotated and reset the saved offset.

        As to your original approach with replacing strings, this also can be done, but this requires opening file in "read-write" mode open(FILE, "+<mylog.file"). Then you would mark position of the string which you want to replace, then seek to that position using seek function and then write the new string. Though you should be careful not to change the length of the string, otherwise you'll corrupt other data as well. After you are done writing, you would seek again to the position where you stopped reading and continue.

        As you can see, that approach is somewhat complex, plus it is inefficient, because you'll have to scan through the same areas multiple times. The only benefit here is automatic handling of log rotation.

Re: Search logs with crontab
by clueless newbie (Friar) on Nov 18, 2013 at 17:18 UTC

    Read the saved tell if any. Seek to that point if it exists. Perform your scan.

    When your scan terminates, save the tell for the next go-a-round. You won't need to replace.

Re: Search logs with crontab
by VincentK (Beadle) on Nov 19, 2013 at 17:03 UTC
    This is overkill, but I think there are some pieces here that you can use. I used temporary and semaphore files.
    use strict; use warnings; use Fcntl qw(:flock); use Time::localtime; $| = 1; my $logFile = "mainLog.LOG"; my $SEMAPHORE = '~'. $logFile . '.lck'; my $was_awaiting = 0; sub WRITE_TO_LOGFILE(); sub READ_LOGFILE_FOR_SENDMAIL (); sub IS_FILE_LOCKED($); sub TIMESTAMP (); ################################################################### # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ################################################################### print "\nWRITE TO LOGFILE .... \n"; WRITE_TO_LOGFILE (); print "\nWRITE TO LOGFILE .... [DONE]\n"; print "\nREAD LOGFILE FOR SENDMAIL .... \n"; READ_LOGFILE_FOR_SENDMAIL (); print "\nREAD LOGFILE FOR SENDMAIL .... [DONE]\n"; ################################################################### # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ################################################################### sub WRITE_TO_LOGFILE () { open(S, ">$SEMAPHORE") or die "$SEMAPHORE: $!"; print "[Semaphore open]\n"; while ( IS_FILE_LOCKED($SEMAPHORE) ) { print "\r\t[Awaiting Semaphore lock]"; $was_awaiting = 1; } if ($was_awaiting) { print "\n"; $was_awaiting = 0; } print "\t[Semaphore lock] \n"; flock(S, LOCK_EX) or die "flock() failed for $SEMAPHORE: $!"; print "\t\t[Logfile open]\n"; open (FH, ">>$logFile") or die "Can't open $logFile: $!"; print FH "I have written from PID ($$)::[".TIMESTAMP() +."]::PHY-4-EXCESSIVE_ERRORS\n"; close FH; print "\t\t[Logfile closed]\n"; print "\t\tGoing to sleep...\n"; sleep 10; print "\t\tWoken up...\n"; close S; print "[Semaphore unlock]\n"; print "[Semaphore closed]\n"; unlink($SEMAPHORE); } sub READ_LOGFILE_FOR_SENDMAIL () { open(S, ">$SEMAPHORE") or die "$SEMAPHORE: $!"; print "[Semaphore open]\n"; while ( IS_FILE_LOCKED($SEMAPHORE) ) { print "\r\t[Awaiting Semaphore lock]"; $was_awaiting = 1; } if ($was_awaiting) { print "\n"; $was_awaiting = 0; } print "\t[Semaphore lock] \n"; flock(S, LOCK_EX) or die "flock() failed for $SEMAPHORE: $!"; print "\t\t[Logfile open]\n"; open (FH, "<$logFile") or die "Can't open $logFile: $!"; open (FHT, ">", '~'."$logFile") or die "Can't open temp $l +ogFile: $!"; while(<FH>) { chomp; if ($_ =~ m/PHY-4-EXCESSIVE_ERRORS/) { ## SENDMAIL() $_ =~ s/PHY-4-EXCESSIVE_ERRORS/PHY_4_EXCESSIVE +-ERRORS -- SENDMAIL SENT/g; print FHT "$_\n"; } else { print FHT "$_\n"; } } close FH; close FHT; unlink($logFile); ## DELETE PREVIOUS LOGFILE rename('~'."$logFile",$logFile); ## RENAME TEMP LOG +FILE TO REPLACE PREVIOUS LOG FILE print "\t\t[Logfile closed]\n"; close S; print "[Semaphore unlock]\n"; print "[Semaphore closed]\n"; unlink($SEMAPHORE); } sub IS_FILE_LOCKED ($) { my $theFile; my $theRC; ($theFile) = @_; $theRC = open(my $HANDLE, ">>", $theFile); $theRC = flock($HANDLE, LOCK_EX|LOCK_NB); close($HANDLE); return !$theRC; } sub TIMESTAMP () { my $t = localtime; return sprintf( "%04d-%02d-%02d_%02d-%02d-%02d", $t->year + 1900, $t->mon + 1, $t->mday, $t->hour, $t->min, $t->sec ); } __END__ -RUNTIME- C:\monks>perl loggie.pl WRITE TO LOGFILE .... [Semaphore open] [Semaphore lock] [Logfile open] [Logfile closed] Going to sleep... Woken up... [Semaphore unlock] [Semaphore closed] WRITE TO LOGFILE .... [DONE] READ LOGFILE FOR SENDMAIL .... [Semaphore open] [Semaphore lock] [Logfile open] [Logfile closed] [Semaphore unlock] [Semaphore closed] READ LOGFILE FOR SENDMAIL .... [DONE] C:\monks> - Log File - I have written from PID (7892)::[2013-11-19_11-52-41]::PHY_4_EXCES +SIVE-ERRORS -- SENDMAIL SENT I have written from PID (7452)::[2013-11-19_11-53-33]::PHY_4_EXCES +SIVE-ERRORS -- SENDMAIL SENT

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1063138]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (9)
As of 2014-08-01 09:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (258 votes), past polls