Need to efficiently merge 2 FIX protocol log files

[softwareCEO]

I run a software firm in financial services and we use the FIX protocol. I have a straight forward coding problem. Can you provide the best sample code to my problem? The FIX protocol is tag value pair protocol with 1 FIX message per line. Every tag value pair is separated by ASCII SOH. Every FIX message ends with 10=xxx<SOH> where xxx is 3 digits. After this, the time stamp exists. I want to merge these 2 files into a 3rd file based on the time stamp on each line. The files could be fairly large (10,000,000 lines, 1 message/line, 100 bytes/message). The speed of the merge is very important. Here are 2 simple sample input files.

file 1:

8=FIX.4.2
9=0059
35=A
49=X
56=Y
52=20121008-12:01:27
34=1
98=0
108=30
10=125
10/08/12 08:01:27.489799

8=FIX.4.2
9=0047
35=0
49=X
56=Y
52=20121008-12:01:51
34=2
10=078
10/08/12 08:01:51.489969

file 2:

8=FIX.4.2
9=63
35=A
34=1
49=B
52=20121008-12:01:27.490
56=A
98=0
108=30
10=227
10/08/12 08:01:27.489930

8=FIX.4.2
9=51
35=0
34=2
49=B
52=20121008-12:01:57.490
56=A
10=186
10/08/12 08:01:57.490432

[NetWallah]

A quick search shows that the Finance::FIX module is a (very) rudimentary parser for the FIX format.

Hopefully, it can get you started . Unfortunately, the module has no parsing for the Timestamp - which seems to be outside of the FIX protocol message.

Since your posting was not formatted properly (See Writeup Formatting Tips), it is near impossible to see distinguishing charecteristics of your message timestamps, which we need to see, in order to propose parsing methods.

[Riales]

Aren't CEOs supposed to pay employees to do this sort of thing?

[CountZero]

Are the FIX files already sorted on timestamp?

What is the format of the date-part of the timestamp? dd/mm/yy or mm/dd/yy or yy/mm/dd or ...?

[space_monk]

Okay, lets assume you have enough memory to hold the entire contents of both files in RAM; the simplest is probably to create a hash with the timestamp as a key, and sort on that key.

However, this doesn't take into account the fact that your original files are probably already sorted, so your algorithm should probably be to compare the leading row of each file, and then write the one with the oldest timestamp value, until you run out of data. Obviously such code has to take into account the possibility of identical timestamps, one file being shorter than another .... yada yada yada.

This is roughly how to do it - it is not completely debugged Perl as I'm short on time

sub compare_ts {
  my ($a, $b) = @_;
  # this function needs changing to compare timestamp strings.
  if ($a < $b) { return -1; }
  elsif ($a == $b) { return 0;}
  elsif ($a > $b) { return 1; }
}

my @els;
my $tsA, $tsB;

# open the files...
open my $fa, "<fileA" or die something;
open my $fb, "<fileB" or die something;

# slurp....
my @aFIX = <$fa>;
my @bFIX = <$fb>;

# prime each compare value
my $rowA = shift @aFIX;
my $rowB = shift @bFIX;

# keep comparing till one or the other runs out of data

while (defined $rowA and defined $rowB) {
    # get timestamps from rows, could use regex: m/10=\d\d\d\x01(\w+)
    # but split on SOH and using last element probably does the job
    @els = split( /\x01/, $rowA);
    $tsA = pop @els;
    @els = split( /\x01/, $rowB);
    $tsB = pop @els;

    if (compare_ts( $tsA, $tsB) < 0) {
        print $rowA; $rowA = shift @aFIX;
    } else {
        print $rowB; $rowB = shift @bFIX;
    }
}

# we've run out of data in fileA or B, so can dump the rest
if (defined $rowA) {
   print $rowA; print @aFIX;
}
if (defined $rowB) {
   print rowB; print @bFIX;
}