Uniq and sort in perl

slayedbylucifer has asked for the wisdom of the Perl Monks concerning the following question:

I am fetching two columns from oracle DB using the DBD::Oracle module. this is my code whihc Actually fetches the output:

my $sth = $dbh->prepare($query);
$sth->execute();

while ( my $row_ref = $sth->fetchrow_arrayref )
{
        my $vm = $row_ref->[0];
        my @nfs = split (/:/, $row_ref->[2]);
        my $filer = $nfs[0];
        print "$vm, $filer, \n";

}
[download]

I have omitted rest of the code but above while loop should get you the idea what I am trying to do. "$query" value was given to me by the Oracle DBA and it is fairly complicated as it does quite a few comparisons. BTW, i dont udnertand how that query is wirtten. Anyway, it fetcehs the information that I need.

The problem is, the output of above "while" loop gives me few duplicate entries as below:

ows301.dom.com, 10.157.118.42,
oem300.dom.com, 10.157.124.58,
omo300.dom.com, 10.157.124.58,
pwd302.dom.com, 10.157.126.58,
pwd302.dom.com, 10.157.126.58,
pwd302.dom.com, 10.157.126.58,
pwd302.dom.com, 10.157.126.58,
pwd302.dom.com, 10.157.126.58,
[download]

Pelase note that the Query statement has the "unique" function running. But after I run the "split" command in while loop, I get above few duplicate entries.

So, How to sort these entries and make them unique. I mean above, above duplicate entries should look like below:

ows301.dom.com, 10.157.118.42,
oem300.dom.com, 10.157.124.58,
pwd302.dom.com, 10.157.126.58,
[download]

please guide me on a technique which can be used in above "while" Loop. In bash, it is fairly simple to run ...| sort | uniq. but I could not find a way in perl to do the same

-Thanks.

Comment on Uniq and sort in perl Select or Download Code

Replies are listed 'Best First'.
Re: Uniq and sort in perl by Utilitarian (Vicar) on Jul 12, 2012 at 08:59 UTC
Any time you need to get a unique list in Perl you should think of using a hash. Say you defined a hash called `%seen` outside the while loop and changed the print line to read... `print "$vm, $filer, \n" if ! $seen{"$vm:$filler"}++;` [download] This will print the entry the first time and increment the seen flag for that pair, now subsequent identical results will encounter a set flag and not print as a result `print "Good ",qw(night morning afternoon evening)[(localtime)[2]/6]," fellow monks."`	[reply] [d/l] [select]
Re^2: Uniq and sort in perl by slayedbylucifer (Scribe) on Jul 12, 2012 at 09:22 UTC
Utilitarian, I can't thank you enough. Your solution is exactly what I was looking for. Thank you very Much !! But, unfortunately i am not able to decode or understand your oneliner. Could you guide me to some documentation for the same. Thanks Again for the lightning fast response.	[reply]
Re^3: Uniq and sort in perl by Anonymous Monk on Jul 12, 2012 at 10:51 UTC
See Re: Uniq and sort in perl, its FAQ You can also see ddg://site:perlmonks.org perlfaq4 uniq	[reply]
Re^3: Uniq and sort in perl by Preceptor (Deacon) on Jul 15, 2012 at 11:26 UTC
What is happening in that one liner: `print "$vm, $filer, \n" if ! $seen{"$vm:$filler"}++` [download] It's quite an elegant piece of code, with a lot going on. So let's break it down: The hash called '%seen' is being checked - to see if it contains a key called "$vm:$filler". The value of this is being tested for being 'false'. (because there's 'if !' which is 'if not') - which is literally 'if not seen' So if $seen{"$vm:filler"} is _not_ 'true' - we haven't seen it, and so we print it. And then that ++ kicks in, to post increment, 'adding one' to $seen{"$vm:$filler"}' - so the first time we try doing that, it's going to be undefined (and thus false) triggering a print. Second time, it'll be 'non zero' and so evaluate as 'true'. For bonus points, you can probably do: `foreach my $key ( keys %seen ) { print "$key was seen $seen{$key} times\n"; }` [download] Does that make it any clearer?	[reply] [d/l] [select]
Re: Uniq and sort in perl by johngg (Canon) on Jul 12, 2012 at 09:40 UTC
Utilitarian has given you a method of printing unique entries but if you also want to sort them then further steps will be required. Instead of printing in your `while` loop, declare an array outside of the loop and push your lines onto it inside the loop. You can then print your sorted unique lines afterwards. I use the `inet_aton()` from Socket to translate the IP numbers to a sortable string and use a Schwartzian Transform to do the sorting. use strict; use warnings; use Socket; my @lines = split m{(?<=\n)}, <<EOD; ows301.dom.com, 10.157.118.42, oem300.dom.com, 10.157.124.58, omo300.dom.com, 10.157.124.58, pwd302.dom.com, 10.157.126.58, pwd302.dom.com, 10.157.126.58, pwd302.dom.com, 10.157.126.58, pwd302.dom.com, 10.157.126.58, pwd302.dom.com, 10.157.126.58, EOD print do { my %seen; map { $_->[ 0 ] } sort { $a->[ 2 ] cmp $b->[ 2 ] \|\| $a->[ 1 ] cmp $b->[ 1 ] } map { my @flds = split m{,\s*}; [ $_, $flds[ 0 ], inet_aton( $flds[ 1 ] ) ] } grep { not $seen{ $_ } ++ } @lines }; [download] The output. `ows301.dom.com, 10.157.118.42, oem300.dom.com, 10.157.124.58, omo300.dom.com, 10.157.124.58, pwd302.dom.com, 10.157.126.58,` [download] I hope this is helpful. Cheers, JohnGG	[reply] [d/l] [select]
Re^2: Uniq and sort in perl by slayedbylucifer (Scribe) on Jul 12, 2012 at 09:56 UTC
Johngg, thanks for the response. your solution is way more detailed and I will definitely refer it in complex scripts where I would need sort and uniq...Thanks.	[reply]
Re: Uniq and sort in perl by Anonymous Monk on Jul 12, 2012 at 08:55 UTC
FAQ it up ! $ perldoc -q duplicate perlfaq4 How can I remove duplicate elements from a list or array?	[reply]
Re: Uniq and sort in perl by thargas (Deacon) on Jul 12, 2012 at 14:41 UTC
You might want to consider whether the SQL is correct or not. If it's supposed to return unique info, the fact that it doesn't sounds suspicious to me. While the various other answers may allow you to see only unique info, this may be covering up a more basic problem.	[reply]
Re^2: Uniq and sort in perl by Utilitarian (Vicar) on Jul 12, 2012 at 15:24 UTC
This is a very valid point, selecting only the fields you are interested in would improve performance as well as being the "correct" way to achieve this `print "Good ",qw(night morning afternoon evening)[(localtime)[2]/6]," fellow monks."`	[reply] [d/l]
Re^3: Uniq and sort in perl by slayedbylucifer (Scribe) on Jul 12, 2012 at 16:58 UTC
No. the SQL query is not giving me the duplicate entries. They look duplicate only after I run the "split" function inside my while loop. Thanks for your time.	[reply]
Re^4: Uniq and sort in perl by CountZero (Bishop) on Jul 13, 2012 at 06:02 UTC


P is for Practical
	PerlMonks