Rather than reply to just one of the two replies that suggest doing your own buffering in Perl, I'll reply to that concept here.

Perl does some unusual buffer management that can actually make Perl's I/O faster than I/O written in C using <stdio.h>. Unfortunately, this buffer management gets into the guts of stdio.h and so can only be done if your stdio.h data structures are pretty "normal".

The command "perl -V:d_stdstdio" is supposed to report "d_stdstdio='undef'" if Configure couldn't verify that it was safe for Perl do these optimizations and so Perl's I/O is not "fast".

If Perl's I/O is "fast", then the underlying C code is already doing the "read a big buffer and split it up" and doing it more than twice as fast as you could do it yourself in your Perl script. But if Perl's I/O isn't "fast", then you might be able to get nearly a two-fold speed-up by buffering and splitting yourself.

I'd previously been told that Linux fails this verification and so Perl's I/O is more than twice as slow as it would be otherwise (nearly four times a slow?). Checking v5.6.1 on Linux, I find d_stdstdio='define', so I'm not sure if Perl has improved on this or something else is going on.

I'd also heard that Win32 doesn't have fast Perl I/O but it also reports d_stdstdio='define'. So perhaps that test isn't very useful. /:

So it certainly may be worth trying the buffer-and-split in Perl and see if that makes your I/O faster or slower. It is rather sad that Perl w/o "faster" I/O doesn't manage to do buffer-and-split in C as fast as can be done in a Perl script. I think that case was considered "rare" and so didn't get much attention.