http://www.perlmonks.org?node_id=91572

pharaoh has asked for the wisdom of the Perl Monks concerning the following question:

I have an array that is comprised of a boat load of ip addresses, many of them duplicates. Is there an easy way to detect and remove all duplicate instances? Thanks in advance.

Replies are listed 'Best First'.
(jeffa) Re: Removing Duplicates in an array?
by jeffa (Bishop) on Jun 26, 2001 at 16:58 UTC
[reply]
      Is it just me, or does the FAQ title 'How can I extract just the unique elements of an array?' not actually convey what the solution does accurately?

      The unique elements of ("fish","bread","eggs","eggs","fish","butter") are ("bread","butter").

      And as a quick golf question: how would you extract the genuinely unique elements of a list?

      Tiefling

      -----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GAT d++ s:- a-- C++ UL P++ L++(+) E? W+(++) N+ o? K w+(--)
!O M- V? PS+ PE- Y PGP- t+ 5 X+ R+++ tv- b+++ DI++++ D+ G+ e++ h!(-) y
+?
------END GEEK CODE BLOCK------

      [download]
[reply]
[d/l]
[reply]
[d/l]
        At 2818 chars, I golf: 
        sub u{@h{@_}=@_;keys%h} 
 
print join "\t", u("fish","bread","eggs","eggs","fish","butter")
#butter  bread   fish    eggs

        [download]
        /me wonders how soon MeowChow will have a shorter solution...

        Jeroen
        "We are not alone"(FZ)
        Update: Too bad. The shortest I can come with is just a rewrite of Davorg's code (he's a golfer in disguise): 

        sub u{$h{$_}++for@_;grep$h{$_}<2,keys%h}

        [download]
[reply]
[d/l]
[select]
        My shortest is 30 chars: 
        sub u {
  grep{$a=$_;2>grep$_ eq$a,@_}@_
}

        [download]
           MeowChow                                   
               s aamecha.s a..a\u$&owag.print
[reply]
[d/l]
Re: Removing Duplicates in an array?
by voyager (Friar) on Jun 26, 2001 at 17:00 UTC
    From the Perl Cookbook: 
    my @list = qw(some list of of non unique words);
my %unique = ();
foreach my $item (@list) {
    $unique{$item}++;
}

    [download]
    The keys of %unique are the unique values.
[reply]
[d/l]
[reply]
[d/l]
        TIMTOWTDI: How about one that preserves the original order? 
          @a=qw( one two three for five six seven eight nine ten eleven);
  @hash{@a}=(0..$#a);
  @sorted=sort { $hash{$a}<=>$hash{$b} } keys %hash;

        [download]

        mr.nick ...

[reply]
[d/l]

        nice solution, but you could still be doing a lot of unnecessary data copying - depending on what is stored in @array - so:

        my %unique;
@unique{@list} = ();

@list = keys %unique;

        [download]

        With the cookbook method you have of course the benefit of being able to get a count of the number of occurences ...

        -- Hofmator

[reply]
[d/l]
Re: Removing Duplicates in an array?
by the_0ne (Pilgrim) on Jun 26, 2001 at 17:35 UTC
    I found this about a month ago, but not sure where. I think it was in a POD, but not really sure though. You have to first make sure the array is sorted... 
    @array = sort { $a cmp $b } @array;

# Now remove dups.
%saw = ();
@de_duped_array = grep (!$saw{$_}++, @array);

    [download]
    That works for me.

    Good luck.
[reply]
[d/l]
[reply]
        Thanks davorg, you're right. I was thinking of a previous version I had tried of that statement that didn't use a hash in the way it is using here.
[reply]
      Ok, just went to my home node and I noticed that I had a link to the 'How can I extract just the unique elements of an array?' That IS where I found the example. Sorry bout the clutter.
[reply]
Re: Removing Duplicates in an array?
by ChemBoy (Priest) on Jun 26, 2001 at 20:26 UTC

    Assuming you want to retain a spanning set (which the golf seems to have gotten away from...) 

    @filtered = grep {!$seen{$_}++} @original;
#somewhat tested code

    [download]
    will give you one of each IP address in the list.

    And it golfs pretty well, too...


    If God had meant us to fly, he would *never* have give us the railroads.
        --Michael Flanders

[reply]
[d/l]
Re: Removing Duplicates in an array?
by Sifmole (Chaplain) on Jun 26, 2001 at 16:54 UTC
    This question gets answered probably once a day around here. Do search on your title and you will come up with plenty of responses.
[reply]

Back to Seekers of Perl Wisdom