in reply to Optimise file line by line parsing, substitute SPLIT
If you were to launch multiple copies of the program in parallel with one another (e.g. just use trailing-& on the Unix shell command-line to start with ...), how much time is added to the overall process if, say, 5 files were processed at the same time? 10?
My guess is that you can get away with that, and if so, the 8-second difference in time becomes a lot less critical to the completion of the business task. (Using split really is more clear ...) There will be some sort of “sweet spot” where the completion-time is about the same ... then a rather abrupt shift (a knee-shaped bend in the performance curve ... a thrash-point) after which completion-times will fall through the floor. But you can fool-around with this idea, using just the command line job facilities of Unix/Linux. I”d love to know if I guessed right.
|
|---|