in reply to
Re^3: selecting columns from a tab-separated-values file
in thread selecting columns from a tab-separated-values file
I'm back at work and have tested the two-process solution. It took 60 seconds to pass 10M (M=million) records. Then I pulled the logic for splitting and joining the records out of obuf and into ibuf (thus eliminating obuf) and ran the same test, and it ran in 62 seconds. (In both cases the output was to /dev/null.)<\p>
I reran the tests sending output to an actual file in the same directory as the input, and obtained exactly the same runtimes.
In ALL cases I observed the CPU of the process that was doing the split/join to peg at 100%.
So I have to conclude that disk I/O is negligible for this program, in my environment.