http://www.perlmonks.org?node_id=1052413

vihar has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am trying to remove duplicate lines from a file if I find a specific string match at a specific location. I am supposed to be opening up a set of 8 files. These files are huge. I am reading them line by line and looking for a unique number at 10th and 11th position on each line. If this number is repeated in the next line, I am supposed to disregard that line. Later, I am connecting to a DB and performing some XML parsing on all feeds that come back from DB for each line. For example, if one of the text file is like this:
AB000000026JHAHKDFK AB000000028JHKHKHKJ AB00000003033AFSFAS AB000000030HJHKH80J AB000000030LOIKJUJ8 AB0000000324446KJHK
I am only supposed to be considering following since 30 is repeated for 2 more lines.
AB000000026JHAHKDFK AB000000028JHKHKHKJ AB00000003033AFSFAS AB0000000324446KJHK
Here is my code:
use DBI; use Time::localtime; use File::Compare; use XML::Simple; # qw(:strict); use Data::Dumper; $user = "213256"; @fileRead = glob '/export/home/$user/Tests/Match/dummy2*'; my @array1; foreach $file (@fileRead){ open(FILE, $file) or die "Can't open `$file': $!"; @lines = <FILE>; close FILE; foreach $line ( @lines ) { $str = $line; $var = substr($str, 10, 2); push(@array1, "$var"); my @unique = grep { ! $seen{$_}++ } @array1; } ...... }
I am stuck in that last foreach loop as I am not sure what I can do. Like I mentioned, I am performing other functions to these lines. But I am just having trouble with avoiding these lines with repeated unique codes. I want this code to be able to move over to next line if same unique code is found.

Thanks