removing repeated elements from one array

[perlbeginner10]

Hi guys, I am beginner in Perl, and would you please help me in this. I have a very long array, and I want it to be unique array. There must be a simple way cut off the duplicated elements. Thanks

[g0n]

From perlfaq4:

Use a hash. When you think the words "unique" or "duplicated", think "hash keys". If you don't care about the order of the elements, you could just create the hash then extract the keys. It's not important how you create that hash: just that you use keys to get the unique elements.

   my %hash   = map { $_, 1 } @array;
   # or a hash slice: @hash{ @array } = ();
   # or a foreach: $hash{$_} = 1 foreach ( @array );

   my @unique = keys %hash;

(This is also cited in perlmonks Q&A here).

[Akhasha]

Note that this works only for scalars (good enough for most purposes) - if its a list of object references and you want duplicates removed:

my %hash = map { $_ => $_ } @array;
# then...
my @unique_unordered = values %hash;
my @unique_ordered = grep {defined} map { delete $hash{$_} } @array;

And for this to make sense the array elements must all be references to objects - if some elements are strings its possible for them to be equal to and overwrite a stringified reference.

[Corion]

Have you used the "Search" box or Super Search to find an answer? Searching for "Removing duplicates", I find Removing Duplicates from an Array and Removing Duplicates in an array?. Searching with Google, the first hit already is an answer. perldoc -q duplicate has the FAQ on that question.

In short, you want to keep track of all items you've already seen, for which a hash is a convenient datastructure:

my %seen;
for my $item (@array) {
  $seen{$item}++ # remember this item
};

my @unique = keys %seen;
# The order gets lost though

[tirwhan]

To preserve order when reducing the list like that, use List::MoreUtils uniq method.

use List::MoreUtils qw(uniq);

my @a=qw(1 3 4 1 2 3);

print join("-",uniq(@a))."\n";

Prints

1-3-4-2

Update:And if you dont't want to install and import an entire module for a simple task like this you could just do what List::MoreUtils does internally:

my %h;
@a=map { $h{$_}++ == 0 ? $_ : () } @a;

(though personally I'd go for using the module in most cases, it's more self-documenting).

[ambrus]

See this recent thread: removing duplicate entries from an array.

[kulls]

Hi,
you can use Array::Unique.

    use Array::Unique;
    tie @arr_name, 'Array::Unique';

where @arr_name will store only unique values,you no need to bother about validation.
-kulls

[vishi83]

Hi perlbeginner10 !!




I'm sure you wud have got your question answered by many people. But, i would hav been very happy if you had searched it yourself, instead of getting help from others. Do you see a search box on the top left corner of perlmonks. Jus type your queries there and you'll get watever you need. Thats how i learnt all these stuffs and you know wat i hav reached 'level 4' in jus 1 month's time..

So, do lot of self-learning and use this group more effectively




http://perlmonks.com/index.pl?node=removing+duplicate+entires&go_button=Search
Thanks

[spiritway]

Another possibility, if you want to keep this as an array. Sort the array. Then iterate through it, comparing consecutive elements. If equal, delete one and repeat, until you've reached the last element.