Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

usage of awk and grep together

by raghu_shekar (Novice)
on May 20, 2009 at 04:57 UTC ( #765108=perlquestion: print w/replies, xml ) Need Help??
raghu_shekar has asked for the wisdom of the Perl Monks concerning the following question:


can someone please tell me how can i use an awk command and a grep command together using pipes and backtics.

for example:
`awk '/fileid/,/^-----/' logfile | grep specificdata`
when i use this in a script i get the below error :
sh[2]: 0403-057 Syntax error at line 2 : `|' is not expected.

Replies are listed 'Best First'.
Re: usage of awk and grep together
by juster (Friar) on May 20, 2009 at 05:36 UTC

    Anything awk/grep can do perl can do better! You don't need to inefficiently call them externally, perl already has adopted their strengths.

    sub get_log_data { open my $logfile, '<', 'logfile' or die "open on logfile: $!"; my @data_lines; while (<$logfile>) { next unless ( /fileid/ .. /^-{5}/ ); # range operator! if (/specificdata/) { push @data_lines, $_; } } close $logfile; die 'could not find specificdata in logfile' unless (@data_lines); return join q{}, @data_lines; }

    Please try to enclose your code and error message in <c> and </c> tags in the future. Your error did not display right and the code is easier to read with these tags.

    Update: fixed bug where only one line of specificdata matching would be returned, not multiple lines.

      Anything awk/grep can do perl can do better!
      Suggesting a 15 line sub as alternative for a short one liner doesn't present anything in support of your claim.
      Hi, Thanks, but im very very new to perl. the fileid and the specific data in the code are also stored in an array. i pick them from an array store them in a variable and substitute them in the command. Also i am not familiar with shell scripting.

        I would suggest playing with creating subs and passing them arguments if you haven't already. Like this example you can take code you already have and put them in a nice friendly named container. This is a fundamental skill that is very handy.

        Call the sub with just the $fileid and $specificdata that you want given as arguments.

        You retrieve the arguments very easily:

        sub get_log_data { my ($fileid, $match) = @_; #...

        Then just replace fileid with $fileid and specificdata with $match.

        As you can see there are many other ways to do this. I provided the longest way I could think of because I hoped it would be easier to understand and learn from. Short one-liners generally have many subtle concepts that make them work. i.e. here is the sub using johngg's approach, one I would use for a quick n dirty script.

        sub get_log_data { my ($fileid, $match) = @_; local @ARGV = 'logfile'; return join '', grep { (/$fileid/o .. /\A-{5}/) && /$match/o } <>; }

        I'd suggest using a long, explicit, easy to debug, easy to understand version for a beginner; as a learning exercise.

Re: usage of awk and grep together
by johngg (Abbot) on May 20, 2009 at 09:13 UTC

    juster's advice to do the job in Perl is sound and the task need not take many lines of code.

    use strict; use warnings; my @wantedLines = grep { m{^fileid} .. m{^-----} and m{pattern} } <>; print for @wantedLines;

    Here is a pretend log file.

    $ cat spw765108.log a line another line fileid 123 some data a pattern we want some more data another wanted pattern yet more data ----- another line a pattern outside our bounds last line $

    Now run the script, giving the log file as an argument.

    $ ./spw765108 spw765108.log a pattern we want another wanted pattern $

    The <> reads the filehandle automatically opened on the file (or files) supplied as arguments to the script. The built-in grep filters the lines read from the filehandle to pass into @wantedLines only those that satisfy the conditions set. The first condition uses the range operator (also known as the flip-flop operator) to select only those lines between the two conditions. A further test to see if any lines match the desired pattern also has to be true.

    I hope this is helpful.



Re: usage of awk and grep together
by ELISHEVA (Prior) on May 20, 2009 at 09:35 UTC

    What operating system are you running this perl script on? What shell is being used by that operating system? When I run your command on using backtics on my system (Debian Linux, bash shell) I don't get an error at all.

    If you need portability across operating systems, I recommend you stay away from backtics and pipes and instead use Perl functions. The following short script does the same thing as your grep/awk combination:

    #strictures - put these two lines at the top of every Perl script #these do a lot of the debugging work for you use strict; use warnings; ... lots of other stuff ... # do: awk '/fileid/,/^-----/' logfile | grep specificdata my $bPrint=0; open LOGFILE, '<', "Monks/tmp/logfile.txt" or die; while (my $line = <LOGFILE>) { #equivalent to awk '/fileid/,/^-----/' logfile $bPrint = 1 if $line =~ /fileid/; $bPrint = 0 if $bPrint && $line =~ /^-----/; #equivalent to grep specific data print $line if $bPrint && $line =~ /specificData/; } close LOGFILE;

    To learn more about the Perl concepts involved in the above code:

    • strict and warnings explain what the two lines at the beginning mean and what they can do for you
    • perlopentut - will explain how to open and close files in Perl
    • retut - will explain how to use =~ to filter data using regular expressions. Perl regular expressions are much more powerful than the awk regular expressions with which you are probably familiar.
    • perlport - things to watch out for if you are using Perl on multiple operating system to other.

    Best, beth

Re: usage of awk and grep together
by Bloodnok (Vicar) on May 20, 2009 at 10:58 UTC
    Hmmm ,

    A coupla things:

    • The error message appears to be AIX in origin ... and I'm not able to reproduce your problem on AIX 5.3.
    • In this case, there's no need to use both awk and grep, one or the other will do.
    The long and short of it is that since grep doesn't directly support ranges, you could condense the above to achieve (what I believe to be) the same result using awk as follows:
    awk '/fileid/,/^-----/ { /specificdata/ { print } }' logfile

    A user level that continues to overstate my experience :-))

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://765108]
Approved by targetsmart
[marioroy]: LA, yeah. than there's no reason why it cannot work inside qx. But chatting is hard in PM. I cannot see the code now.
[shmem]: Lady_Aleena: sometimes a quick glance isn't enough.
[Lady_Aleena]: What?! Can't stat ~/Music: No such file or directory? The ~/ doesn't expand?
[Lady_Aleena]: And the env variable $HOME doesn't work either. !@#$

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (10)
As of 2017-04-23 21:00 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (432 votes). Check out past polls.