Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

log parser

by alex_fatu (Novice)
on Jun 17, 2013 at 07:36 UTC ( #1039284=perlquestion: print w/ replies, xml ) Need Help??
alex_fatu has asked for the wisdom of the Perl Monks concerning the following question:

hi there, monks!

i am new to Perl.
i am a Java, C/C++/C# programmer, recently i also administrate some servers and i need to make a log parser.

let's say my logs size its aprox 2Go. after some test of parsing a simple text i have the next results:

soft powergrep(windows7) - aprox 5 mins
powershell script made by me (new to powershell also) - aprox 5 min
Perl script - aprox 1 min (maximum) - i think..if i will use regexp it will go to 40 sec..maybe.. (today is the first day using Perl, Powershell, scripts)

what i need from this script. Let's say that my log have errors..like DB error ORA-xxxx etc. If i find ORA-xxxx ..maybe print this in a new log and ..the most important: send an e-mail and a text message on my phone.

I need some ideas, please..best practices..i'm sure that i'm not the only one who needs this script.. maybe one of you have something like this...

any info is very useful

btw..this is the script with 1 min search result:

#!/usr/bin/perl -w use strict; use IO::File; use constant FILE => 'D:\ps-scripts\DirectX.log'; use constant FIND => 'eroare'; IO::File->input_record_separator(FIND); my $fh = IO::File->new(FILE, O_RDONLY) or die 'Could not open file ', FILE, ": $!"; $fh->getline; #fast forward to the first match #print each occurence in the file print IO::File->input_record_separator while $fh->getline; $fh->close;

Comment on log parser
Download Code
Re: log parser
by Anonymous Monk on Jun 17, 2013 at 07:50 UTC

    btw..this is the script with 1 min search result:

    It doesn't do any searching at all

    See perlintro, see ack

      The link to "ack" is unfortunately broken.
      What did you mean it to point at, Anonymous Monk?

      Cheers, Sören

      Added later: Thanks for the links, Anonymous Monk

      (hooked on the Perl Programming language)

        Like some kind of ack or ack -- dang, its not like checking links is hard for a computer :)
      maybe, for me, the word "search" means something else. let me put it this way:

      i tested on a directx log with size=1.8 Go and i manually inserted word "eroare" in some lines.
      when running that script..word "eroare" it's printed on perl cmd if the script finds that word in directx.log.

      thank you.
      Alex

        maybe, for me, the word "search" means something else. let me put it this way:

        :) My turn

        input_record_separator separates "records" , like by default it does "lines" , you can't use it to parse logs, you can't use it to search

        For searching you need index/rindex/Regular expressions

      y u no like grep?
Re: log parser
by ultibuzz (Monk) on Jun 17, 2013 at 09:05 UTC

    i noticed that you mentioned ORA-xxx error codes
    if you want to parse Oracle logs, check inside your OracleDB directory, there are a bunch of utility and log perl scripts

    kind regards
    Alex

Re: log parser
by alex_fatu (Novice) on Jun 17, 2013 at 10:06 UTC
    can anyone tell me what i do wrong here:
    #!/usr/bin/perl use strict; use warnings; print "searching using regexp..."; open(my $in, "<", "DirectX.log") or die "Can't open DirectX.log: $! +"; #while (<$in>) { # assigns each line in turn to $_ # print "Just read in this line: $_"; #} if ($in =~ /eroare/) { print "found eroare\n"; } close $in or die "$in: $!";

      you do a regex on your filehandler and not it content :)
      try

      #!/usr/bin/perl use strict; use warnings; print "searching using regexp..."; open(my $in, "<", "DirectX.log") or die "Can't open DirectX.log: $! +"; while (<$in>) { # assigns each line in turn to $_ print "Just read in this line: $_"; if ($_ =~ /eroare/) { print "found eroare on line nr: $.\n"; } } close $in or die "$in: $!";

        thank you!!!!
      oke, i found the problem.

      the next code print a msg if word "eroare" is found:

      #!/usr/bin/perl use strict; use warnings; print "searching using regexp...\n"; my $a = "start"; open(my $in, "<", "DirectX.log") or die "Can't open DirectX.log: $! +"; #print $a; while (<$in>) { # assigns each line in turn to $_ #print "Just read in this line: $_"; $a = $a . $_; } if ($a =~ /eroare+/) { print "found eroare\n"; } close $in or die "$in: $!";


      can anyone tell me how can i make it to print every occurance of the match?

        Now you read the whole file into $a (which can be done more efficiently) but what you want is something like

        while (<$in>) { # assigns each line in turn to $_ #print "Just read in this line: $_"; if( /eroare/ ) { # checks $_ print; } }

        check the while loop usage:

        while(my $line=<$in>){ if ($line=~/eroare/i){ print "Found"; } }

        If you tell me, I'll forget.
        If you show me, I'll remember.
        if you involve me, I'll understand.
        --- Author unknown to me
Re: log parser
by alex_fatu (Novice) on Jun 17, 2013 at 12:51 UTC
    thank you all!

    now i have another problem. if the word is found, i need to send an email.

    on perl.org/exemples i have found the following code:

    #!/usr/bin/perl use strict; use warnings; # first, create your message use Email::MIME; my $message = Email::MIME->create( header_str => [ From => 'you@example.com', To => 'friend@example.com', Subject => 'Happy birthday!', ], attributes => { encoding => 'quoted-printable', charset => 'ISO-8859-1', }, body_str => "Happy birthday to you!\n", ); # send the message use Email::Sender::Simple qw(sendmail); sendmail($message);


    and i have implemented it in my code as:

    #!/usr/bin/perl use strict; use warnings; use Email::MIME; print "searching using regexp...\n"; open(my $in, "<", "DirectX.log") or die "Can't open DirectX.log: $! +"; while (<$in>) { if ($_ =~ /eroare+/) { print "found eroare\n"; my $message = Email::MIME->create( header_str => [ From => 'alexandru.fatu@hotmail.com', To => 'alexandru.fatu@hotmail.com', Subject => 'Error!', ], attributes => { encoding => 'quoted-printable', charset => 'ISO-8859-1', }, body_str => "Eroare gasita: $_", ); # send the message use Email::Sender::Simple qw(sendmail); sendmail($message); } } close $in or die "$in: $!";


    After downloading Email::MIME and all dependencies (email::address, email::sample and many more..) i get an error:
    unable to establish SMTP connection

    Trace begun at C:\strawberry\perl\site\lib\Email\Sender\Transport\SMTP +.pm line 9 6 Email::Sender::Transport::SMTP::_throw('Email::Sender::Transport::SMTP +=HASH(0x1e 3e90c)', 'unable to establish SMTP connection') called at C:\strawberr +y\perl\sit e\lib\Email\Sender\Transport\SMTP.pm line 63 Email::Sender::Transport::SMTP::_smtp_client('Email::Sender::Transport +::SMTP=HAS H(0x1e3e90c)') called at C:\strawberry\perl\site\lib\Email\Sender\Tran +sport\SMTP .pm line 105 Email::Sender::Transport::SMTP::send_email('Email::Sender::Transport:: +SMTP=HASH( 0x1e3e90c)', 'Email::Abstract=ARRAY(0x1d4132c)', 'HASH(0x1ca4bec)') ca +lled at C: \strawberry\perl\site\lib\Email\Sender\Role\CommonSending.pm line 27 Email::Sender::Role::CommonSending::__ANON__ at C:\strawberry\perl\ven +dor\lib\Tr y\Tiny.pm line 76 eval {...} at C:\strawberry\perl\vendor\lib\Try\Tiny.pm line 67 Try::Tiny::try('CODE(0x1e471a4)', 'Try::Tiny::Catch=REF(0x1e31c14)') c +alled at C :\strawberry\perl\site\lib\Email\Sender\Role\CommonSending.pm line 40 Email::Sender::Role::CommonSending::send('Email::Sender::Transport::SM +TP=HASH(0x 1e3e90c)', 'Email::Abstract=ARRAY(0x1d4132c)', 'HASH(0x1e4503c)') call +ed at C:\s trawberry\perl\site\lib\Email\Sender\Simple.pm line 115 Email::Sender::Simple::send_email('Email::Sender::Simple', 'Email::Abs +tract=ARRA Y(0x1d4132c)', 'HASH(0x1d3d77c)') called at C:\strawberry\perl\site\li +b\Email\Se nder\Role\CommonSending.pm line 27 Email::Sender::Role::CommonSending::__ANON__ at C:\strawberry\perl\ven +dor\lib\Tr y\Tiny.pm line 76 eval {...} at C:\strawberry\perl\vendor\lib\Try\Tiny.pm line 67 Try::Tiny::try('CODE(0x1be23ec)', 'Try::Tiny::Catch=REF(0x1d2fb8c)') c +alled at C :\strawberry\perl\site\lib\Email\Sender\Role\CommonSending.pm line 40 Email::Sender::Role::CommonSending::send('Email::Sender::Simple', 'Ema +il::MIME=H ASH(0x1d4127c)') called at C:\strawberry\perl\vendor\lib\Sub\Exporter\ +Util.pm li ne 18 Sub::Exporter::Util::__ANON__('Email::MIME=HASH(0x1d4127c)') called at + bebe1.pl line 35


    Please, tell me where do i need to put my email, my password etc..setting for sending an email.

    BR,
    Alex

      check Net::SMTP::OneLiner Module
      and define your not changing variables like email, sender, host, etc. outside the while loop so you do not define them over and over.

      After downloading Email::MIME and all dependencies (email::address, email::sample and many more..) i get an error: unable to establish SMTP connection

      The example code you found assumes that the local host running your script has an MTA (mail transport agent) such as sendmail installed locally.

      The documentation for Email::Sender shows how to use Email::Sender::Transport for communicating with a remote SMTP server.

      -- FloydATC

      Time flies when you don't know what you're doing

Re: log parser
by pklausner (Scribe) on Jun 17, 2013 at 14:40 UTC
    If you want a really fast search, look at Sean O’Rourke’s Perl code which won the widefinder parallel search performance contest.

    But as you are into monitoring, check out the brazillion scripts which do that. Nagios' check_logfiles may be a bit basic, but then there are swatch, logwatch or SEC if you want to do really fancy correlation on your logs. Plus there should be Oracle specific scanners, Nagios plug-ins, whatnot...

    If your primary mission is not to learn Perl, then there is no need to write lowly grep loops...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1039284]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (14)
As of 2014-10-24 11:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (131 votes), past polls