Re: Removing duplicates lines

[sundialsvc4]

Your first-attempt code tries to do the impossible:   to read the entire file into memory.   Fortunately, you don’t need to do that.

The filter that you seek to build only ever needs to consider two lines:   “this one,” and “the previous one if any.”   Thus, you can read and process the file (or files ...) line-by-line with an algorithm that looks something like this:

my $previous_line = undef;  # INITIALLY, THERE IS NO 'PREVIOUS LINE'
while ( my $current_line = <FILE> ) {
   next if 
      defined($previous_line) &&
      (substr($current_line, 10, 2) eq substr(previous_line, 10, 2) ) 
+ 
       # OR WHATEVER-IT-IS ...
      ;

   << "ooh, we survived!!" so, do something magical >>

   $previous_line = $current_line;
}

Perl’s short-circuit boolean evaluation comes in handy here... the special case of “this is the first line in the file” is marked by $previous_line = undef, and the condition as-written expressly omits that case from consideration, evaluating the substr()s only when both values are known to exist.