Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Syslog files revisited

by johngg (Abbot)
on Aug 02, 2012 at 22:38 UTC ( #985108=note: print w/ replies, xml ) Need Help??


in reply to Syslog files revisited

split and sprintf will probably do what you want.

knoppix@Microknoppix:~$ perl -Mstrict -Mwarnings -E ' > my $line = q{May 2 04:06:15 lon-pop.mail.mydom.com pop3login: LOGOU +T, user=gonenow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent= +5560, time=1}; > my ( $mon, $day, $time, $dom, $login, $remainder ) = > split m{:?\s+}, $line, 6; > my %monthNos = do { > my $no = 0; > map { $_ => ++ $no } > qw{ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec }; > }; > > my $yr = q{2012}; > my $csv = sprintf q{%02d/%02d/%s %s,%s,%s,%s}, > $day, $monthNos{ $mon }, $yr, $time, $dom, $login, $remainder; > > say $csv;' 02/05/2012 04:06:15,lon-pop.mail.mydom.com,pop3login,LOGOUT, user=gone +now, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=5560, time=1 knoppix@Microknoppix:~$

I hope this is helpful.

Update: After replying to your reply I realised I had totally missed the need to quote the variable message and add the user and ip to the csv line. Here is revised code.

knoppix@Microknoppix:~$ perl -Mstrict -Mwarnings -E ' > my $line = q{May 2 04:06:15 lon-pop.mail.mydom.com pop3login: LOGOU +T, user=gonenow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent +=5560, time=1}; > my ( $mon, $day, $time, $dom, $login, $remainder ) = > split m{:?\s+}, $line, 6; > my %monthNos = do { > my $no = 0; > map { $_ => ++ $no } > qw{ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec }; > }; > > my $yr = q{2012}; > my ( $user, $ip ) = > $remainder =~ m{user=([^,]+),\s+ip=\[([^\]]+)}; > $remainder = qq{"$remainder"}; > my $csv = sprintf q{%02d/%02d/%s %s,%s,%s,%s,%s,%s}, > $day, $monthNos{ $mon }, $yr, $time, $dom, > $login, $remainder, $user, $ip; > > say $csv;' 02/05/2012 04:06:15,lon-pop.mail.mydom.com,pop3login,"LOGOUT, user=gon +enow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=5560, time= +1",gonenow,::ffff:127.0.0.1 knoppix@Microknoppix:~$

Cheers,

JohnGG


Comment on Re: Syslog files revisited
Select or Download Code
Re^2: Syslog files revisited
by stevbutt (Novice) on Aug 02, 2012 at 23:15 UTC

    This is certainly a move in the right direction and the date part is superb !, but it gives me the problem that the variable message at the end gets split up into an indeterminate number of csv values where if it were at least quoted I could read that as one field as the commas inside the quotes would be ignored ( I think )

      ... but it gives me the problem that the variable message at the end gets split up into an indeterminate number of csv values ...

      No, the third parameter to split limits the number of resultant fields so the $remainder scalar variable holds the entirety of your variable message.

      Cheers,

      JohnGG

        Thats great and thank you very much for your help. I have one further issue though. My existing code now looks like this

        #!/usr/bin/perl use 5.010; use strict; use warnings; while (my $line = <STDIN>) { chomp($line); my ( $mon, $day, $time, $dom, $login, $remainder ) = split m{:?\s+}, $line, 6; my %monthNos = do { my $no = 0; map { $_ => ++ $no } qw{ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec }; }; my $yr = q{2012}; my $csv = sprintf q{%02d/%02d/%s %s,%s,%s,"%s"}, $day, $monthNos{ $mon }, $yr, $time, $dom, $login, $remainder; say $csv; }

        As I wanted the remainder to be in double quotes so it would read in as one field. The problem is that remainder sometimes already contains double quotes which I would like to remove or replace with single quotes

        I tried the following but it leaves be with an empty set of double quotes

        my $remainder =~ s/""\"\""/\'/g;
Re^2: Syslog files revisited
by stevbutt (Novice) on Aug 10, 2012 at 13:51 UTC

    Thanks John, Almost all lines are now being loaded correctly but I have one further problem with occasional lines and specifically just on the user= part.

    Sometimes it says this

    user=gonenow sent=34 recv=34

    Or

    user=<gonenow>, etc etc

    Instead of the comma separated list I expected

    Is there a way to just extract my user value up until either a comma or space and to strip < > if they occur ?

      The user=<gonenow>, etc etc case can probably be got around with a couple of minor changes to the code. The user=gonenow sent=34 recv=34 is more of a problem as it fails the premise on which the split approach was based. I'd probably wrap the original code in an if condition and deal with the non-comma-separated lines in an else clause.

      It just so happens that I am about to go on holiday so I will not be able to provide further help for a week or so. I suggest you create another question in SoPW, linking to this thread, and post some example data lines illustrating your problem so that others can perhaps help you build on the solution you have so far.

      Cheers,

      JohnGG

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://985108]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2014-12-23 05:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (135 votes), past polls