Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

[CountZero]

For the "simple" case where you have all files in strict time-sequential order:

use Modern::Perl;

open my $FILEA, '<', './FILEA.TXT' or die $!;
open my $FILEB, '<', './FILEB.TXT' or die $!;
open my $FILEC, '<', './FILEC.TXT' or die $!;
open my $FILED, '<', './FILED.TXT' or die $!;

my $ts_B = 0;
my $ts_C = 0;
my $ts_D = 0;
while ( my $ts_A = <$FILEA> ) {
    chomp $ts_A;
    say "A:$ts_A matches B:$ts_B" if $ts_A == $ts_B;
    while ( $ts_B = <$FILEB> ) {
        chomp $ts_B;
        last if $ts_B > $ts_A;
        next if $ts_B < $ts_A;
        say "A:$ts_A matches B:$ts_B";
    }
    say "A:$ts_A matches C:$ts_C" if $ts_A == $ts_C;
    while ( $ts_C = <$FILEC> ) {
        chomp $ts_C;
        last if $ts_C > $ts_A;
        next if $ts_C < $ts_A;
        say "A:$ts_A matches C:$ts_C";
    }
    say "A:$ts_A matches D:$ts_D" if $ts_A == $ts_D;
    while ( $ts_D = <$FILED> ) {
        chomp $ts_D;
        last if $ts_D > $ts_A;
        next if $ts_D < $ts_A;
        say "A:$ts_A matches D:$ts_D";
    }
}

Output

A:102 matches B:102
A:264 matches C:264
A:403 matches D:403
A:403 matches D:403
A:560 matches D:560
A:560 matches D:560
A:744 matches B:744
A:744 matches B:744
A:827 matches C:827
A:915 matches D:915
A:915 matches D:915
A:1018 matches C:1018
A:1180 matches D:1180
...

The files I used don't have timestamps, just random numbers in an ever increasing sequence, but the logic remains the same.

[Cosmic37]

Thank you all so much for your comments and guidance. Thank you to those who pointed out the very important caveat that certain logic requires monotonically increasing time within the datafiles - on this occasion this is the case thankfully.

During my experimentation I had a lot of problems with the following and I am guessing some of you will be able to tell me why as I am once again eeegnorant! The error I get is only at the second match even though $fha and $fhb get set as expected if I print them to screen. The error reported is "Use of uninitialized value in pattern match (m//) at whatever.pl line whatever, <INPUTB> line whatever". I suppose I also need to chomp the input lines before concatenating them but the point is that the match is failing and I can't see why... Many thanks in advance for any suggestions.

...
open OUTPUT, $outputFile
    or die "Can't open $outputFile for writing $!\n";
open INPUTA, $inputFileA
    or die "Can't open $inputFileA for writing $!\n";
open INPUTB, $inputFileB
    or die "Can't open $inputFileB for writing $!\n";
while($fha=<INPUTA>){
    print "a: ",$fha;
    if($fha=~m/(\d{2}):(\d{2}):(\d{2}).+(\d{2})(\d{2})(\d{2})\d{2}_\d{
+2}\.abc/){
    $timea=timegm($3,$2,$1,$4,$5,$6);
    $output_line=$fha;
    until($timeb>$timea){
        $fhb=<INPUTB>;
        print "b: ",$fhb;
        if($fhb=~m/(\d{2}):(\d{2}):(\d{2}).+(\d{2})(\d{2})(\d{2})\d{2}
+_\d{2}\.abc/){
        $timeb=timegm($3,$2,$1,$4,$5,$6);
        if($timeb=$timea){
            $output_line.=$fhb;
        }
        }
    }
    print OUTPUT $output_line,"\n";
    }
}
close OUTPUT;
close INPUTA;
close INPUTB;

[Cosmic37]

I'm imagining :-D that each file opened has a pointer or equivalent indicating which line we are at. Then if we make a match on the first line of fileA the "while ( $ts_B = <$FILEB> )" statement is not scanning from the beginning of FILEB but from where it previously left off when we are seeking a match for the second line of FILEA. Is this correct? This is obviously important for efficiency and speed as I don't want to be starting all over and scanning through the whole file for each timestamp.

[CountZero]

Indeed, each filehandle is independent of all others and will keep its own reference to where it was in the file it is connected to. So it has no need to rescan from the very beginning of the file to get the next line.