Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

How to extract duplicate lines from a text file?

by rnaeye (Pilgrim)
on Mar 01, 2011 at 05:27 UTC ( #890680=perlquestion: print w/ replies, xml ) Need Help??
rnaeye has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am a geneticist/biochemist and trying to use Perl to analyze DNA sequence data. I have a giant text file (99 million lines; ~ 7GB). The file contains DNA sequences. I need to extract duplicate lines based on the BeadIDs from input file and print it into different file. BeadIDs are written in field one (first column), for example followings are BeadIDs:

99_99_992 99_999_930 99_999_930 99_99_999 99_99_999 99_99_9992 99_99_9992 99_99_9992

Real Input file looks like this (total ~99 million lines):

99_99_992 chr16.fsa 11064 11088 TAGGTAAACGAGGAGTCTTTCAATA + - 25 99_999_930 chr04.fsa 148776 148800 TGCAAACGAGGACAACCTGTTTG +TG + 25 99_999_930 chr04.fsa 148882 148916 CCGGAAAAATTTGCTATTGGAAG +AGGTGGCGCTGG - 35 99_99_999 chr12.fsa 468017 468049 TTTTCGGTGACGGAAATACGCTTC +AGAGACCCT + 33 99_99_999 chr12.fsa 468138 468162 CTATGTTTTCTTACTCCTATGTCT +A - 25 99_99_9992 chr12.fsa 468138 468162 CTATGTTTTCTTACTCCTATGTC +TA - 25 99_99_9992 chr12.fsa 468138 468162 CTATGTTTTCTTACTCCTATGTC +TA - 25 99_99_9992 chr12.fsa 468138 468162 CTATGTTTTCTTACTCCTATGTC +TA - 25 ...

The output that I would like to have is shown below (BeadIDs must be repeated exactly TWICE, not unique, not triplicate, not quadruplicate, but I want the entire line not only column one):

99_999_930 chr04.fsa 148776 148800 TGCAAACGAGGACAACCTGTTTG +TG + 25 99_999_930 chr04.fsa 148882 148916 CCGGAAAAATTTGCTATTGGAAG +AGGTGGCGCTGG - 35 99_99_999 chr12.fsa 468017 468049 TTTTCGGTGACGGAAATACGCTTC +AGAGACCCT + 33 99_99_999 chr12.fsa 468138 468162 CTATGTTTTCTTACTCCTATGTCT +A - 25

I have written a code that would choose duplicate lines if my input file had only single column(field). However, as you can see my actual file has 7 fields, and I need to choose the duplicate lines based on first field, and print the entire line. Sor far, I could come up with following script that would work on a single column file. However, my Perl knowledge is not advance enough to come up for a solution when input file is in 7 field format.

My code so far:

#!/usr/bin/perl #use strict; use warnings; $numArgs = $#ARGV + 1; if ($numArgs < 2) { print "USAGE: extract_duplicate_rows <input file> <# of duplicate +rows>\n\n"; exit(0); } $finput = $ARGV[0]; $duplicate_count = $ARGV[1]; open(IFILE, "<", $finput) or die $!; chomp($prev_line = <IFILE>); $match_count = 1; while (1) { chomp($line = <IFILE>); if ($line eq $prev_line) { $match_count++; } else { if ($match_count == $duplicate_count) { print "$prev_line\n"; } $prev_line = $line; $match_count = 1; if (eof(IFILE)) { last; } } } close IFILE;

I was wondering if you guys could make me suggestions. Thanks for your help. ~Erbay

Comment on How to extract duplicate lines from a text file?
Select or Download Code
Re: How to extract duplicate lines from a text file?
by NetWallah (Abbot) on Mar 01, 2011 at 06:26 UTC
    Try this (more idiomatic) version:
    #!/usr/bin/perl use strict; use warnings; my ($fname, $dupcount) = @ARGV; die "USAGE: extract_duplicate_rows <input file> <# of duplicate rows>\ +n\n" unless $fname and $dupcount; open(my $IFILE, "<", $fname) or die "Cannot open $fname: $! "; my @buf; while (my $line = <$IFILE>) { if (@buf){ if (substr($line,0,10) eq substr($buf[0],0,10)){ push @buf,$line; next }; if (@buf ==$dupcount){ print @buf; }; }; @buf=($line); } @buf==$dupcount and print @buf; close $IFILE;
    This assumes that the input file is sorted.

         Syntactic sugar causes cancer of the semicolon.        --Alan Perlis

      Thanks a lot for the help.

Re: How to extract duplicate lines from a text file?
by Eliya (Vicar) on Mar 01, 2011 at 06:42 UTC

    You can use split to extract individual fields from a line. E.g. something like this:

    ... my $match_count = 1; my $prev_bead_id = ''; my $out = ''; while (my $line = <IFILE>) { my ($bead_id) = split ' ', $line; if ($bead_id eq $prev_bead_id) { $match_count++; } else { if ($match_count == $duplicate_count) { print $out; } $match_count = 1; $out = ''; } $prev_bead_id = $bead_id; $out .= $line; } # for when there is no further line after the last pair in the file: if ($match_count == $duplicate_count) { print $out; }

    Update: fixed boundary case bug.

      Thank you

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://890680]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2014-08-30 22:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls