Re: Remove duplicate entries

[ikegami]

Normalise the key (remove the junk characters) and use it as a hash key.

my %seen;
while (<>) {
   my @fields = ...;
   my $key = normalise($fields[0]);
   print { $seen{$key}++ ? $unique_fh : $extras_fh } $_;
}