Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Script to manipulate data is not giving me any output

by ebi (Initiate)
on Sep 17, 2009 at 09:04 UTC ( #795819=perlquestion: print w/ replies, xml ) Need Help??
ebi has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, I am new to perl and have created the following script which should create an output file ( input.txt ) which I then run against a sh script which creates an ouput which I re direct to another file caled snapvault_status.out. The problem is that the last output file is empty but if I run the command on the command line I do get a filed file. I am sure my code is not great but I just started with perl :-) Any advice is appreciated.
sub read_data { system "ssh datab01 snapvault status | tail -n +3 > /mnt/scri +pts/lagtime/tmp/input.txt"; system "ssh datab02 snapvault status | tail -n +3 >> /mnt/scri +pts/lagtime/tmp/input.txt"; system "ssh datab03 snapvault status | tail -n +3 >> /mnt/scri +pts/lagtime/tmp/input.txt"; print "Data is uitgelezen.\n"; } sub create_correct_output { open(MYINPUTFILE, "</mnt/scripts/lagtime/tmp/input.txt"); while(<MYINPUTFILE>) { cat /mnt/scripts/lagtime/tmp/input.txt |awk '{printf " +%1.50s\t%0.60s\t%0.45s\t %0.45s\n", $1, $2, $3, $4}' | sh /mnt/script +s/lagtime/calculate_lag.sh > /tmp/snapvault_status.out.out; } print "data has been put in the right format.\n"; }
Thanks in advance. Eric

Comment on Script to manipulate data is not giving me any output
Download Code
Re: Script to manipulate data is not giving me any output
by almut (Canon) on Sep 17, 2009 at 09:23 UTC
    while(<MYINPUTFILE>) { cat /mnt/scripts/lagtime/tmp/input.txt |awk '{printf "%1.5 +0s\t%0.60s\t%0.45s\t %0.45s\n", $1, $2, $3, $4}' | sh /mnt/scripts/la +gtime/calculate_lag.sh > /tmp/snapvault_status.out.out; }

    You're calling the awk command (or trying to... you forgot system) for every line in the input file (input.txt) without using that line anywhere in the command ...  What was the idea?

      Awk is needed to put the columns in the right place and then pipe the output to the calculate script which basically looks at a value higher then 24 and then is needs to be placed in the /tmp/snapvault_status.out.out file It looks like I should not use the while statement?
        It looks like I should not use the while statement?

        Yes, I can't see much use in it — except to slow down the processing ;)

        As it looks, calling it just once (using system !) should be sufficient.

Re: Script to manipulate data is not giving me any output
by JavaFan (Canon) on Sep 17, 2009 at 09:28 UTC
    That code doesn't actually compile.

    Nor doesn't make any sense. Why would you want to pass your entire log file through awk for each line of the file? And since you aren't doing anything interesting in Perl, why not write the entire thing in a shell script?

    As for your actual question, you're calling system, but you aren't checking its result. Perhaps an error occurred.

Re: Script to manipulate data is not giving me any output
by swampyankee (Parson) on Sep 17, 2009 at 09:32 UTC

    First suggestion, make sure that your open worked.

    Second is that I believe neither cat nor awk are Perl functions; you have to wrap them in a system call, qx or back quotes (`) for them to be executed. Better would be to translate the awk into Perl (use a2p, if needed) and replace cat, awk, and the shell program with a split, the few lines of Perl to replicate the script's functionality, and printf.


    Information about American English usage here and here. Floating point issues? Please read this before posting. — emc

      thank you very much I will re do the script now with your suggestions. Still learning sorry if the script I wrote made little sense.
Re: Script to manipulate data is not giving me any output
by Bloodnok (Vicar) on Sep 17, 2009 at 11:41 UTC
    Ignoring the fact that the line inside the while loop is not valid perl, if it were in a shell script, I submit that it would be a contender for the Useless cat award since
    cat /mnt/scripts/lagtime/tmp/input.txt |awk '{printf "%1.50s\t%0.60s\t +%0.45s\t %0.45s\n", $1, $2, $3, $4}' | sh /mnt/scripts/lagtime/calcul +ate_lag.sh > /tmp/snapvault_status.out.out;
    is identical to
    awk '{printf "%1.50s\t%0.60s\t%0.45s\t %0.45s\n", $1, $2, $3, $4}' /mn +t/scripts/lagtime/tmp/input.txt | sh /mnt/scripts/lagtime/calculate_l +ag.sh > /tmp/snapvault_status.out.out;
    Moreover, elsewhere, you claim that Awk is needed to put the columns in the right place... - a palpably untrue claim since you're simply using awk to strip out and re-format the first 4 columns, leaving the ordering untouched.

    The problem is that the last output file is empty - this shouldn't come as a huge surprise if the last of the input lines causes an empty line to be written to the output - via the /mnt/scripts/lagtime/calculate_lag.sh script - since, within the loop, the output file is rewritten, c/w being appended to, each time.

    In the light of this, I think I'd be tempted to

    1. Modify /mnt/scripts/lagtime/calculate_lag.sh to take a single line as an argument
    2. Coalesce the ssh invocations with the awk command
    3. Finally, to re-write it [the snippet] something along the following lines i.e. untested:
    use warnings; use strict; use autodie; my @HOSTS = qw/datab01 datab02 datab03/; my $IN_FILE = q(/mnt/scripts/lagtime/tmp/input.txt); my $OUT_FILE = q(/tmp/snapvault_status.out.out); open IN_FILE, qq/>$IN_FILE/; foreach my $host (@HOSTS) { foreach (`ssh $host snapvault status | tail -n +3`) { local @_ = split; # Avoid warnings printf IN_FILE "%1.50s\t%0.60s\t%0.45s\t %0.45s\n", $_[0], $_[ +1], $_[2], $_[3]; } } close IN_FILE; open IN_FILE, qq/<$IN_FILE/; open OUT_FILE, qq/>$OUT_FILE/; while (<IN_FILE>) { print OUT_FILE `sh /mnt/scripts/lagtime/calculate_ +lag.sh $_` } close IN_FILE; close OUT_FILE;
    A user level that continues to overstate my experience :-))
Re: Script to manipulate data is not giving me any output
by ELISHEVA (Prior) on Sep 17, 2009 at 12:10 UTC

    I first began using Perl when an old AWK script became unmanagable. a2p (already cited by swampyankee above) converts AWK scripts to Perl and was tremendously helpful for learning how do things in a Perlish way. I strongly recommend experimenting with it if you are familiar with AWK and are learning Perl.

    Unfortunately, it just dumps the Perl code and doesn't really explain how constructs in AWK and Perl compare, so I've added an annotated example here. Note the difference in order. On the bash command line, you use the order: (a) specify input (b) specify processing (c) specify output. Here we switch (b) and (c), and open both the input and output streams before processing.

    use strict; use warnings; # open an input stream # same effect as: # cat /mnt/scripts/lagtime/tmp/input.txt| # if fail, print reason ($!) and abort open IN, '<', '/mnt/scripts/lagtime/tmp/input.txt' or die "Couldn't open input file: $!"; # open an output stream # same effect as: # > /tmp/snapvault_status.out.out; # if fail, print reason ($!) and abort open OUT, '>', '/tmp/snapvault_status.out.out' or die "Couldn't open output file: $!"; # equivalent to your awk program # { printf "%1.50s\t%0.60s\t%0.45s\t %0.45s\n", $1, $2, $3, $4 } while (my $line=<DATA>) { # chomp: remove record separator # split: break into fields using default AWK field separator # (a single space which is equivalent to the Perl regex m{ } my @aFields = split(m{ }, $line); chomp $line; printf OUT "%1.50s\t%0.60s\t%0.45s\t %0.45s\n", @aFields }

    Also, when you open an input or output stream, always check for errors. There are ever so many reasons why a file doesn't get opened when you think it should. If you use open ... or die ..., you will save yourself hours scratching your head wondering why your program doesn't print anything.

    To learn more about the commands in this script, see open, chomp, and split. You may also find these in depth tutorials on opening files (perlopentut) and regular expressions (perlretut) helpful.

    Best, beth

    Update: Added links to relevant Perl documentation.

Re: Script to manipulate data is not giving me any output
by Damashii (Scribe) on Sep 17, 2009 at 12:40 UTC
    Don't forget
    use strict; use warnings;
      Thank you all very much for your advice this is very helpfull !!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://795819]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (14)
As of 2014-10-22 16:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (119 votes), past polls