Remove duplicate strings from an array

[Anonymous Monk]

Hiya.

OK, maybe im looking in the wrong places or something, I found loads on removing duplicates from arrays but the examples they gave only seemed to work for integers.

One solution is to pop off each element of the array and compare it against every other element remaining. If there's a match, chuck it, otherwise it must be unique so add it to @output. Surely there's a more efficient perlesque way though?

So the idea is if I have an array:

@array = ("abc","def","abc","ghi","ghi","abc","jklm","abc","def")

The output should be:

@output = ("abc","def","ghi","jklm")

Can someone offer a Perl newbie a hand?!

Also can anyone suggest other good sites I can draw code samples/tutorials/information from?

Thanks!

[Corion]

This is a FAQ:

perldoc -q duplicate

How can I remove duplicate elements from a list or array?

Your error likely stems from the fact that you're trying to use == as comparison operator, but == is the operator used to check for numeric comparison. If you want to compare two strings for equality as strings, use the eq operator. But it's better to use the method outlined in the FAQ.

[liverpole]

A really quick way to do it is to create a hash, using the elements of the array as keys.

The keys of a hash are always unique, so you just have to reconstruct the list of keys at the end:

use strict;
use warnings;

my @array  = ("abc","def","abc","ghi","ghi","abc","jklm","abc","def");
my %hash   = map { $_ => 1 } @array;
my @unique = keys %hash;

[davorg]

You should check the Perl FAQ.

How can I remove duplicate elements from a list or array?

[ferreira]

This is almost a FAQ.

sub uniq {
    my %h;
    return grep { !$h{$_}++ } @_
}

[johngg]

This is similar to liverpole's solution but populates the hash using a hash slice rather than a map. Also, you seem to want the resultant array sorted so I've done that.

use strict;
use warnings;

use Data::Dumper;

my @array = qw{abc def abc ghi ghi abc jklm abc def};

my %comb;
@comb{@array} = ();
my @uniq = sort keys %comb;

print Data::Dumper->Dumpxs([\@uniq], [qw{*uniq}]);

The output is

@uniq = (
          'abc',
          'def',
          'ghi',
          'jklm'
        );

I hope this is of use.

Cheers,

JohnGG

[agianni]

This sounds a lot like homework, so rather than giving you a solution, consider looking at the solutions that you said worked with integers and change your equality operators to those that compare strings.

[CountZero]

And using a module (List::MoreUtils):

use List::MoreUtils qw(uniq);
my @x = uniq 1, 1, 2, 2, 3, 5, 3, 4; # returns 1 2 3 5 4

And this function preserves the order in the array (which the hash-based solutions don't).

[Anonymous Monk]

I used perlfaq which suggested some of the code you guys have mentioned. Basically I get an output of the full list again...

Here's my code, the file read in is a csv...

       my %defaultArgs = (
               'porder'   => 1,
               'ptitle1'  => 2,
               'ptitle2'  => 3,
               'ptitle3'  => 4,
               'yorder'   => 5,
               'xorder'   => 6,
               'mtitle'   => 7,
               'size'     => 8,
               'avail'    => 9,
               'select'   => 10,
               'symbol'   => 11,
               'invert'   => 14,
               'isvary'   => 15,
               'unit'     => 16,
               'default'  => 17,
               'min'      => 18,
               'max'      => 19,
               'relative' => 20,
               'monitor'  => 21,
               'ac'       => 22,
               'en'       => 23,
               'summary'  => 24,
               'device'   => 25
           );

my @arguments  = ();
my @devices = ();
my @unique = ();       

@devices = split( ',', $arguments[$defaultArgs{ device }] );

# foreach my $device (@devices)
# {
#    print $device."\n";
# }

my %hash   = map { $_ => 1 } @devices;
my @unique = keys %hash;                

foreach my $uni (@unique)
{
   print $uni."\n";
}