Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Removing Duplicates in an array?

by pharaoh (Initiate)
on Jun 26, 2001 at 16:53 UTC ( #91572=perlquestion: print w/replies, xml ) Need Help??

pharaoh has asked for the wisdom of the Perl Monks concerning the following question:

I have an array that is comprised of a boat load of ip addresses, many of them duplicates. Is there an easy way to detect and remove all duplicate instances? Thanks in advance.

Replies are listed 'Best First'.
(jeffa) Re: Removing Duplicates in an array?
by jeffa (Bishop) on Jun 26, 2001 at 16:58 UTC
      Is it just me, or does the FAQ title 'How can I extract just the unique elements of an array?' not actually convey what the solution does accurately?

      The unique elements of ("fish","bread","eggs","eggs","fish","butter") are ("bread","butter").

      And as a quick golf question: how would you extract the genuinely unique elements of a list?


      -----BEGIN GEEK CODE BLOCK----- Version: 3.1 GAT d++ s:- a-- C++ UL P++ L++(+) E? W+(++) N+ o? K w+(--) !O M- V? PS+ PE- Y PGP- t+ 5 X+ R+++ tv- b+++ DI++++ D+ G+ e++ h!(-) y +? ------END GEEK CODE BLOCK------
        At 2818 chars, I golf:
        sub u{@h{@_}=@_;keys%h} print join "\t", u("fish","bread","eggs","eggs","fish","butter") #butter bread fish eggs
        /me wonders how soon MeowChow will have a shorter solution...

        "We are not alone"(FZ)
        Update: Too bad. The shortest I can come with is just a rewrite of Davorg's code (he's a golfer in disguise):

        sub u{$h{$_}++for@_;grep$h{$_}<2,keys%h}
        My shortest is 30 chars:
        sub u { grep{$a=$_;2>grep$_ eq$a,@_}@_ }
                       s aamecha.s a..a\u$&owag.print
Re: Removing Duplicates in an array?
by voyager (Friar) on Jun 26, 2001 at 17:00 UTC
    From the Perl Cookbook:
    my @list = qw(some list of of non unique words); my %unique = (); foreach my $item (@list) { $unique{$item}++; }
    The keys of %unique are the unique values.
        TIMTOWTDI: How about one that preserves the original order?
        @a=qw( one two three for five six seven eight nine ten eleven); @hash{@a}=(0..$#a); @sorted=sort { $hash{$a}<=>$hash{$b} } keys %hash;

        mr.nick ...

        nice solution, but you could still be doing a lot of unnecessary data copying - depending on what is stored in @array - so:

        my %unique; @unique{@list} = (); @list = keys %unique;

        With the cookbook method you have of course the benefit of being able to get a count of the number of occurences ...

        -- Hofmator

Re: Removing Duplicates in an array?
by the_0ne (Pilgrim) on Jun 26, 2001 at 17:35 UTC
    I found this about a month ago, but not sure where. I think it was in a POD, but not really sure though. You have to first make sure the array is sorted...
    @array = sort { $a cmp $b } @array; # Now remove dups. %saw = (); @de_duped_array = grep (!$saw{$_}++, @array);
    That works for me.

    Good luck.
        Thanks davorg, you're right. I was thinking of a previous version I had tried of that statement that didn't use a hash in the way it is using here.
      Ok, just went to my home node and I noticed that I had a link to the 'How can I extract just the unique elements of an array?' That IS where I found the example. Sorry bout the clutter.
Re: Removing Duplicates in an array?
by ChemBoy (Priest) on Jun 26, 2001 at 20:26 UTC

    Assuming you want to retain a spanning set (which the golf seems to have gotten away from...)

    @filtered = grep {!$seen{$_}++} @original; #somewhat tested code
    will give you one of each IP address in the list.

    And it golfs pretty well, too...

    If God had meant us to fly, he would *never* have give us the railroads.
        --Michael Flanders

Re: Removing Duplicates in an array?
by Sifmole (Chaplain) on Jun 26, 2001 at 16:54 UTC
    This question gets answered probably once a day around here. Do search on your title and you will come up with plenty of responses.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://91572]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2022-05-19 23:53 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (72 votes). Check out past polls.