Let's ask this. Without worrying about efficiency yet, how would you determine the record separator of one of the files? Until I am clear on how you're able to detect the record separator in the first place, I have a difficult time suggesting an efficient means of doing so. What heuristics are you using? Or is the record separator specified in some sort of header for the file? Or does it depend on the file's extension?

If you haven't figured that part out yet, you have to step back from the problem and look at it as a human would. Ask yourself, "If I opened the file in an editor (possibly one that displays non-printables too), how would I spot the record separator?" Once you've figured that out, the next step is to isolate the rules, and put them to code. After you get that working, tests and all, you're done. Only then, if you feel the outcome isn't efficient enough for your needs, should you begin profiling and determining what needs to be made more efficient.

There's an old expression, that Perl is great for prototyping, and often the result is good enough that there's no need to rewrite in C. The same applies here; get it working, and it may be good enough that you don't need to be further concerned with efficiency.