How to extract duplicate lines from a text file?

Hi,

I am a geneticist/biochemist and trying to use Perl to analyze DNA sequence data. I have a giant text file (99 million lines; ~ 7GB). The file contains DNA sequences. I need to extract duplicate lines based on the BeadIDs from input file and print it into different file. BeadIDs are written in field one (first column), for example followings are BeadIDs:

99_99_992
99_999_930
99_999_930
99_99_999   
99_99_999
99_99_9992
99_99_9992
99_99_9992
[download]

Real Input file looks like this (total ~99 million lines):

99_99_992    chr16.fsa    11064    11088    TAGGTAAACGAGGAGTCTTTCAATA 
+   -    25
99_999_930    chr04.fsa    148776    148800    TGCAAACGAGGACAACCTGTTTG
+TG    +    25
99_999_930    chr04.fsa    148882    148916    CCGGAAAAATTTGCTATTGGAAG
+AGGTGGCGCTGG    -    35
99_99_999    chr12.fsa    468017    468049    TTTTCGGTGACGGAAATACGCTTC
+AGAGACCCT    +    33
99_99_999    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTCT
+A    -    25
99_99_9992    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTC
+TA    -    25
99_99_9992    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTC
+TA    -    25
99_99_9992    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTC
+TA    -    25
...
[download]

The output that I would like to have is shown below (BeadIDs must be repeated exactly TWICE, not unique, not triplicate, not quadruplicate, but I want the entire line not only column one):

99_999_930    chr04.fsa    148776    148800    TGCAAACGAGGACAACCTGTTTG
+TG    +    25
99_999_930    chr04.fsa    148882    148916    CCGGAAAAATTTGCTATTGGAAG
+AGGTGGCGCTGG    -    35
99_99_999    chr12.fsa    468017    468049    TTTTCGGTGACGGAAATACGCTTC
+AGAGACCCT    +    33
99_99_999    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTCT
+A    -    25
[download]

I have written a code that would choose duplicate lines if my input file had only single column(field). However, as you can see my actual file has 7 fields, and I need to choose the duplicate lines based on first field, and print the entire line. Sor far, I could come up with following script that would work on a single column file. However, my Perl knowledge is not advance enough to come up for a solution when input file is in 7 field format.

My code so far:

 
#!/usr/bin/perl
#use strict;
use warnings;

$numArgs = $#ARGV + 1;

if ($numArgs < 2)
{
    print "USAGE: extract_duplicate_rows <input file> <# of duplicate 
+rows>\n\n";
    exit(0);
}

$finput = $ARGV[0];
$duplicate_count = $ARGV[1];


open(IFILE, "<", $finput) or die $!;

chomp($prev_line = <IFILE>); 
$match_count = 1;

while (1)
{
    chomp($line = <IFILE>); 
    
    if ($line eq $prev_line)
    {
        $match_count++;
    }
    else
    {
        if ($match_count == $duplicate_count)
        {    
            print "$prev_line\n";
        }

        $prev_line = $line;
        $match_count = 1;

        if (eof(IFILE))
        {
            last;
        }        
    }
}

    
close IFILE;
[download]

I was wondering if you guys could make me suggestions. Thanks for your help. ~Erbay

Comment on How to extract duplicate lines from a text file? Select or Download Code
Re: How to extract duplicate lines from a text file? by NetWallah (Monsignor) on Mar 01, 2011 at 06:26 UTC
Try this (more idiomatic) version: `#!/usr/bin/perl use strict; use warnings; my ($fname, $dupcount) = @ARGV; die "USAGE: extract_duplicate_rows <input file> <# of duplicate rows>\ +n\n" unless $fname and $dupcount; open(my $IFILE, "<", $fname) or die "Cannot open $fname: $! "; my @buf; while (my $line = <$IFILE>) { if (@buf){ if (substr($line,0,10) eq substr($buf[0],0,10)){ push @buf,$line; next }; if (@buf ==$dupcount){ print @buf; }; }; @buf=($line); } @buf==$dupcount and print @buf; close $IFILE;` [download] This assumes that the input file is sorted. Syntactic sugar causes cancer of the semicolon. --Alan Perlis	[reply] [d/l]
Re^2: How to extract duplicate lines from a text file? by rnaeye (Monk) on Mar 01, 2011 at 13:28 UTC
Thanks a lot for the help.	[reply]
Re: How to extract duplicate lines from a text file? by Eliya (Priest) on Mar 01, 2011 at 06:42 UTC
You can use split to extract individual fields from a line. E.g. something like this: `... my $match_count = 1; my $prev_bead_id = ''; my $out = ''; while (my $line = <IFILE>) { my ($bead_id) = split ' ', $line; if ($bead_id eq $prev_bead_id) { $match_count++; } else { if ($match_count == $duplicate_count) { print $out; } $match_count = 1; $out = ''; } $prev_bead_id = $bead_id; $out .= $line; } # for when there is no further line after the last pair in the file: if ($match_count == $duplicate_count) { print $out; }` [download] Update: fixed boundary case bug.	[reply] [d/l]
Re^2: How to extract duplicate lines from a text file? by rnaeye (Monk) on Mar 01, 2011 at 13:30 UTC
Thank you	[reply]


good chemistry is complicated, and a little bit messy -LW
	PerlMonks

How to extract duplicate lines from a text file?

The best material for plates (tableware) is: