How can I keep the first occurrence from duplicated strings?

[Anonymous Monk]

Hi Monks,
if you have a file with lines like the following:

nick 5
nick 10
nick 20
john 78
erik 9
erik 12

Can you tell me how I can keep only the first occurence of each name? If I do a hash for instance, I will keep nick 20, but I want to keep nick 5 instead. How do I do that?

[choroba]

You can still use a hash, but you have to check whether the key already exists before setting the value.

#!/usr/bin/perl
use warnings;
use strict;

my %seen;
while (<DATA>) {
    my ($name, $count) = split;
    $seen{$name} = $count unless exists $seen{$name};

}
print "$_ $seen{$_}\n" for keys %seen;

__DATA__
nick 5
nick 10
nick 20
john 78
erik 9
erik 12

If you want to store the values in an array, use

    push @keep, $_ unless $seen{$name}++;

Or, you can print the lines directly:

    print unless $seen{$name}++;

[GrandFather]

The defined or assignment operator only assigns the right hand value if the value to the left is undefined so you can use that to assign just the first value found to a hash:

use strict;
use warnings;

my $input = <<IN;
nick 5
nick 10
nick 20
john 78
erik 9
erik 12
IN
my %hits;

open my $fin, '<', \$input;

while (my $line = <$fin>) {
    next if $line !~ /^(\w+)\s+(\d+)/;
    $hits{$1} //= $2;
}

print "$_: $hits{$_}\n" for sort keys %hits;

Prints:

erik: 9
john: 78
nick: 5

[tybalt89]

#!/usr/bin/perl

use strict; # https://www.perlmonks.org/?node_id=11154123
use warnings;
use List::AllUtils qw( uniq_by );

open my $fh, '<', \<<END or die;
nick 5
nick 10
nick 20
john 78
erik 9
erik 12
END

my @firstofeachname = uniq_by { (split)[0] } <$fh>;

print @firstofeachname;

Outputs:

nick 5
john 78
erik 9

[tybalt89]

Using hash with //=

#!/usr/bin/perl

use strict; # https://www.perlmonks.org/?node_id=11154123
use warnings;

open my $fh, '<', \<<END or die;
nick 5
nick 10
nick 20
john 78
erik 9
erik 12
END

my %hash;
$hash{ (split)[0] } //= $_ while <$fh>;
print values %hash;

Outputs(order varies by run because of perl's hash randomizing):

nick 5
erik 9
john 78

[eyepopslikeamosquito]

For fun, a more verbose version that you might find easier to understand or maintain:

# print-first.pl. See Perl Monks [id://11154123].
use strict;
use warnings;

my $fname = shift or die "usage: $0 file\n";
open( my $fh, '<', $fname ) or die "error: open '$fname': $!";
my %hash;
my $line;
while ( defined($line = <$fh>) ) {
   chomp $line;
   $line =~ s/^\s+//;                        # remove leading
   $line =~ s/\s+$//;                        # and trailing whitespace
   next unless length $line;                 # ignore empty lines
   next if $line =~ /^#/;                    # ignore comment lines
   next unless $line =~ /^(\w+)\s+(\d+)$/;   # ignore lines that do no
+t match
   my $key = $1;
   my $val = $2;
   next if exists $hash{$key};               # ignore if already seen
   ++$hash{$key};
   print "$key $val\n";
}
close $fh;

Example run:

$ cat t1.txt
nick 5
nick 10
nick 20
john 78
erik 9
erik 12

$ perl print-first.pl
usage: print-first.pl file

$ perl print-first.pl t1.txt
nick 5
john 78
erik 9

See Also

• Why is my code producing weird output? - is this a follow up node by the same anonymonk? (similar file format)
• Long list is long by Chuma - also similar format to this classic old node

Updated: Added "See Also" section

[Bod]

You could just reverse the order of the lines in the file...

use strict;
use warnings;

my @lines = reverse(<DATA>);

my %test;
foreach (@lines) {
    my ($name, $number) = split / /; 
    $test{$name} = $number;
}

print %test;

__DATA__
nick 5
nick 10
nick 20
john 78
erik 9
erik 12

This gives the result...

nick5
john78
erik9

[afoken]

You could just reverse the order of the lines in the file...

use strict;
use warnings;

my @lines = reverse(<DATA>);

This will read the entire file into RAM. No problem for 100 kBytes, big trouble for big files (larger than free RAM). The solutions from choroba, Grandfather, eyepopslikeamosquito, and the second solution from tybalt89 do not suffer from that problem, because they all read only one line at a time.

Alexander

[haukex]

See also File::ReadBackwards (however, its documentation doesn't mention file encodings, so it might blow up on UTF-8).

[Bod]

This will read the entire file into RAM. No problem for 100 kBytes, big trouble for big files

Yes of course that's true.

But given the apparent nature of the data, I feel it's safe to assume that the file size will be small relative to available RAM and paging files. If it were more than a few lines then processing it using Perl is almost certainly the wrong approach. For a file large enough to be a problem, Perl should be reading in one line at a time and loading it into a database when the desired result of getting the first occurrence becomes trivial.

So for a big huge file, the question needs asking on SQLMonks (wishful thinking...)

[Discipulus]

Hello,

many variations but at glance it seems to me that this is missing: use a hash and put vualues into an array (choroba suggested this..).

Being an opportunity to produce a oneliner, I cannot resist to :)

cat uniqfirst.txt

nick 5
nick 10
nick 20
john 78
erik 9
erik 12

perl -lanE "push@{$r{$F[0]}},$F[1]}{say join' ',$_,$r{$_}[0]for keys%r
+" uniqfirst.txt

erik 9
nick 5
john 78


perl -MO=Deparse -lanE "push@{$r{$F[0]}},$F[1]}{say join' ',$_,$r{$_}[
+0]for keys%r"

BEGIN { $/ = "\n"; $\ = "\n"; }
use feature 'current_sub', 'evalbytes', 'fc', 'postderef_qq', 'say', '
+state', 'switch', 'unicode_strings', 'unicode_eval';
LINE: while (defined($_ = readline ARGV)) {
    chomp $_;
    our @F = split(' ', $_, 0);
    push @{$r{$F[0]};}, $F[1];
}
{
    say join(' ', $_, $r{$_}[0]) foreach (keys %r);
}
-e syntax OK

L*