Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

by CountZero (Bishop)
on Sep 22, 2013 at 12:46 UTC ( #1055182=note: print w/ replies, xml ) Need Help??


in reply to matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

For the "simple" case where you have all files in strict time-sequential order:

use Modern::Perl; open my $FILEA, '<', './FILEA.TXT' or die $!; open my $FILEB, '<', './FILEB.TXT' or die $!; open my $FILEC, '<', './FILEC.TXT' or die $!; open my $FILED, '<', './FILED.TXT' or die $!; my $ts_B = 0; my $ts_C = 0; my $ts_D = 0; while ( my $ts_A = <$FILEA> ) { chomp $ts_A; say "A:$ts_A matches B:$ts_B" if $ts_A == $ts_B; while ( $ts_B = <$FILEB> ) { chomp $ts_B; last if $ts_B > $ts_A; next if $ts_B < $ts_A; say "A:$ts_A matches B:$ts_B"; } say "A:$ts_A matches C:$ts_C" if $ts_A == $ts_C; while ( $ts_C = <$FILEC> ) { chomp $ts_C; last if $ts_C > $ts_A; next if $ts_C < $ts_A; say "A:$ts_A matches C:$ts_C"; } say "A:$ts_A matches D:$ts_D" if $ts_A == $ts_D; while ( $ts_D = <$FILED> ) { chomp $ts_D; last if $ts_D > $ts_A; next if $ts_D < $ts_A; say "A:$ts_A matches D:$ts_D"; } }
Output
A:102 matches B:102 A:264 matches C:264 A:403 matches D:403 A:403 matches D:403 A:560 matches D:560 A:560 matches D:560 A:744 matches B:744 A:744 matches B:744 A:827 matches C:827 A:915 matches D:915 A:915 matches D:915 A:1018 matches C:1018 A:1180 matches D:1180 ...
The files I used don't have timestamps, just random numbers in an ever increasing sequence, but the logic remains the same.

CountZero

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

My blog: Imperial Deltronics


Comment on Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles
Select or Download Code
Re^2: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles
by Cosmic37 (Novice) on Sep 22, 2013 at 16:40 UTC

    Thank you all so much for your comments and guidance. Thank you to those who pointed out the very important caveat that certain logic requires monotonically increasing time within the datafiles - on this occasion this is the case thankfully.

    During my experimentation I had a lot of problems with the following and I am guessing some of you will be able to tell me why as I am once again eeegnorant! The error I get is only at the second match even though $fha and $fhb get set as expected if I print them to screen. The error reported is "Use of uninitialized value in pattern match (m//) at whatever.pl line whatever, <INPUTB> line whatever". I suppose I also need to chomp the input lines before concatenating them but the point is that the match is failing and I can't see why... Many thanks in advance for any suggestions.

    ... open OUTPUT, $outputFile or die "Can't open $outputFile for writing $!\n"; open INPUTA, $inputFileA or die "Can't open $inputFileA for writing $!\n"; open INPUTB, $inputFileB or die "Can't open $inputFileB for writing $!\n"; while($fha=<INPUTA>){ print "a: ",$fha; if($fha=~m/(\d{2}):(\d{2}):(\d{2}).+(\d{2})(\d{2})(\d{2})\d{2}_\d{ +2}\.abc/){ $timea=timegm($3,$2,$1,$4,$5,$6); $output_line=$fha; until($timeb>$timea){ $fhb=<INPUTB>; print "b: ",$fhb; if($fhb=~m/(\d{2}):(\d{2}):(\d{2}).+(\d{2})(\d{2})(\d{2})\d{2} +_\d{2}\.abc/){ $timeb=timegm($3,$2,$1,$4,$5,$6); if($timeb=$timea){ $output_line.=$fhb; } } } print OUTPUT $output_line,"\n"; } } close OUTPUT; close INPUTA; close INPUTB;
Re^2: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles
by Cosmic37 (Novice) on Sep 22, 2013 at 17:21 UTC

    I'm imagining :-D that each file opened has a pointer or equivalent indicating which line we are at. Then if we make a match on the first line of fileA the "while ( $ts_B = <$FILEB> )" statement is not scanning from the beginning of FILEB but from where it previously left off when we are seeking a match for the second line of FILEA. Is this correct? This is obviously important for efficiency and speed as I don't want to be starting all over and scanning through the whole file for each timestamp.

      Indeed, each filehandle is independent of all others and will keep its own reference to where it was in the file it is connected to. So it has no need to rescan from the very beginning of the file to get the next line.

      CountZero

      A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

      My blog: Imperial Deltronics

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1055182]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (7)
As of 2014-12-25 19:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (162 votes), past polls