Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

String searching in file

by cbtshare (Scribe)
on Feb 09, 2018 at 19:35 UTC ( #1208859=perlquestion: print w/replies, xml ) Need Help??
cbtshare has asked for the wisdom of the Perl Monks concerning the following question:

Hello All, I am getting a list of ipaddresses from aws into an array then search if that ip is already in the file, if not insert it, but having a bit of trouble with the grep command, I also tried sed but the logic doesnt work, am I doing it wrong? I don't want to send the command results to a file, then open and read that file, search that way for the ip, then input input into another file. So at first awsList is blank, then command runs, nothing should be found, so it inserts all the ips, then on subsequent runs if it finds ip in awsLists, it does nothing

#!/usr/bin/perl use warnings; use strict; my $awsLists = '/ansible/awsLists'; my @command; @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" | grep PrivateDnsNam$ foreach my $line (@command) { chomp($line); if (grep /$line/, $awsLists ) { print "found\n"; } else { print "not found\n"; system qq{ echo "$line" >> "$awsLists"}; } }
if ( ! `sed "/$line/p" $awsLists` ) { print "not found\n"; system qq{ echo "$line" >> "$awsLists"}; } else { print "found\n"; }

I also tried grep -Fxq $line $awsLists

Replies are listed 'Best First'.
Re: String searching in file
by poj (Monsignor) on Feb 09, 2018 at 20:34 UTC
    if (grep /$line/, $awsLists )

    Using perl, grep on the lines in the file not the filename. For example (using test data instead of your command output)

    #!/usr/bin/perl use warnings; use strict; my $awsLists = '/ansible/awsLists'; my @awsLists = (); if (-e $awsLists){ open my $fh,'<',$awsLists or die "$!"; @awsLists = <$fh>; close $fh; } my $cmd = 'aws ec2 describe-instances --filters "Name=instance-state-n +ame,Values=running" | grep PrivateDnsNam$'; my @output = <DATA>;#qx/$cmd/; open my $fh,'>>',$awsLists or die "$!"; foreach my $line (@output){ chomp($line); if (grep /$line/, @awsLists ){ print "$line found\n"; } else { print "$line not found - added\n"; print $fh $line."\n"; } } close $fh; __DATA__ test 1 test 89
    poj
      Thank you very much, I see what you have done, checking if the file exists, opening it for reading, then opening it again for writing.I was kinda trying to avoid that, and just skip those extra steps and search if it ip address is in the file to begin with, if not write to it. Is there no way to do that, without opening the file for reading and writing?
      Sorry but looking at your code more closely, I dont see how it will work, sorry if I didnt put all the details. $awsLists isn't populated, so my goal is to populate the file, then check if ip is there, and not do anything, but if it is there write to the file
      @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" | grep PrivateDnsName | cut -d ":" -f2| cut -d '" +' -f2 | sort -u `;
      Result
      ip-10-1-68-84.us-west-2.compute.internal ip-10-1-7-183.us-west-2.compute.internal ip-10-1-80-15.us-west-2.compute.internal ip-10-1-80-162.us-west-2.compute.internal ip-10-1-80-184.us-west-2.compute.internal ip-10-1-80-187.us-west-2.compute.internal ip-10-1-80-18.us-west-2.compute.internal ip-10-1-80-57.us-west-2.compute.internal
Re: String searching in file
by Laurent_R (Canon) on Feb 09, 2018 at 22:07 UTC
    I think that the best way is to load your file into a hash (each IP as a key, the hash value doesn't matter), close the file; then add each IP in your array to the hash (a hash will remoce duplicates anyway); once you've finished that, just write the hash keys (i.e. unique IP addresses) to a file (or to the same file if you're confident that it works OK).

    A quick untested example:

    open my $IN, "<", $awsLists or die "could not open $awsLists $!"; my %hash = map { chomp; $_ => 1 } <$IN>; close $IN; chomp @command; # just in case you need it. Note: you could choose a b +etter name for your array $hash{$_} = 1 for @command; open my $OUT, ">", "$awsLists.out" or die "could not open $awsLists.ou +t $!"; print "$_\n" for keys %hash; close $OUT;
    Using a hash lookup will be usually be much faster than greping your file or its content many times.
      thanks, will test it out.
Re: String searching in file
by tybalt89 (Priest) on Feb 10, 2018 at 00:05 UTC

    Here's one similar to Re: String searching in file with a couple of extra features.

    1. If file does not exist, that's OK, will make one.
    2. Leaves \n on ip names for ease of handling.
    3. Only writes file if new ips are added.

    Note: filename and @commands have to be fixed, I changed them for testing purposes.

    #!/usr/bin/perl # http://perlmonks.org/?node_id=1208859 use strict; use warnings; my $awsLists = 'd.1208859'; # change to your filename my @commands = <DATA>; # new ips (with \n) my %old; if( open my $in, '<', $awsLists ) # OK if file not there { @old{ <$in> } = (); # note: keys have \n close $in; } my $oldsize = keys %old; @old{ @commands } = (); # add new ips if( keys %old > $oldsize ) # only write if new ips added { open my $out, '>', $awsLists or die "$! opening $awsLists"; print $out sort keys %old; # sort not needed, but nice touch :) close $out; } __DATA__ ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-168.us-west-2.compute.internal ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-158.us-west-2.compute.internal ip-10-1-129-78.us-west-2.compute.internal ip-10-1-1-145.us-west-2.compute.internal ip-10-1-65-215.us-west-2.compute.internal ip-10-1-129-194.us-west-2.compute.internal ip-10-1-1-252.us-west-2.compute.internal ip-10-1-67-133.us-west-2.compute.internal
      thank you very much, my final code and explanation is below, please let me know if I am not understanding something
      use warnings; use strict; my $line; my $awsLists = '/ansible/awsLists'; my @command; @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" | grep PrivateDnsName | cut -d ":" -f2| cut -d '" +' -f2 | sort -u `; ##Initializes an empty hash my %old; ##Checks if the file is there to be opened and read into the handle $i +n if( open my $in, '<', $awsLists ) # OK if file not there { ##If file is there then read contents into keys and associating a valu +e of blank or () means new line? ## why wouldn't you do @old{<$in>} = undef; , would it be wrong? @old{ <$in> } = (); # note: keys have \n close $in; } ###This takes the keys from the the hash previously populate from the +file and puts it into scalar my $oldsize = keys %old; ## Populates the hash again, but this time with the values of the @com +mand array ## Does the old keys get overwritten?(I dont think so, eg, an array + values dont get overwritten everytime you push new values) @old{ @command } = (); # add new ips #Does the compare, and if the hash keys from the <$in> and different f +rom the hash keys of @command if( keys %old > $oldsize ) # only write if new ips added { open my $out, '>', $awsLists or die "$! opening $awsLists"; ##print directly to the file the new keys difference print $out sort keys %old; # sort not needed, but nice touch :) close $out; } # system qq{sed -i '/^ *\$/d' $awsLists};

        ###This takes counts the keys from the the in the hash previously populate from the file and puts it into scalar

        my $oldsize = keys %old;

        #Does the compare, and if the count of hash keys from the <$in> and different from the hash keys of @command has increased (i.e. new keys added)

        if( keys %old > $oldsize ) # only write if new ips added

        ## If file is there then read contents into keys and associating a value of blank or () means new line?
        ## why wouldn't you do @old{<$in>} = undef; , would it be wrong?

          @old{ <$in> } = (); # note: keys have \n

        In this case no difference but possibly misleading you to think @old{<$in>} = 1; would assign the value 1 to all the keys from <$in> which is not the case. Try this simple example

        #!/usr/bin/perl use strict; use Data::Dumper; my @keys = ('a','b','c'); my %hash=(); @hash{@keys} = 1; print Dumper \%hash; __END__ $VAR1 = { 'c' => undef, 'a' => 1, 'b' => undef };
        poj
Re: String searching in file
by marinersk (Priest) on Feb 09, 2018 at 20:08 UTC

    I don't see your results in the post; nor what was expected and why it's different.

    I see an opening backtick on the @command= line, but no closing backtick. Is that normal?

      sorry about that:
      @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" | grep PrivateDnsName | cut -d ":" -f2| cut -d '" +' -f2 | sort -u `;

      Sample Results in @command:

      ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-168.us-west-2.compute.internal ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-158.us-west-2.compute.internal ip-10-1-129-78.us-west-2.compute.internal ip-10-1-1-145.us-west-2.compute.internal ip-10-1-65-215.us-west-2.compute.internal ip-10-1-129-194.us-west-2.compute.internal ip-10-1-1-252.us-west-2.compute.internal ip-10-1-67-133.us-west-2.compute.internal
      My issue is that it keeps on printing "not found" even when the ip address is in the file

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1208859]
Approved by herveus
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (1)
As of 2018-05-26 00:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?