Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: Optimise file line by line parsing, substitute SPLIT

by sundialsvc4 (Abbot)
on Jun 03, 2013 at 13:26 UTC ( #1036747=note: print w/replies, xml ) Need Help??

in reply to Optimise file line by line parsing, substitute SPLIT

If you were to launch multiple copies of the program in parallel with one another (e.g. just use trailing-& on the Unix shell command-line to start with ...), how much time is added to the overall process if, say, 5 files were processed at the same time?   10?

My guess is that you can get away with that, and if so, the 8-second difference in time becomes a lot less critical to the completion of the business task.   (Using split really is more clear ...)   There will be some sort of “sweet spot” where the completion-time is about the same ... then a rather abrupt shift (a knee-shaped bend in the performance curve ... a thrash-point) after which completion-times will fall through the floor. But you can fool-around with this idea, using just the command line job facilities of Unix/Linux.   I”d love to know if I guessed right.

  • Comment on Re: Optimise file line by line parsing, substitute SPLIT

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1036747]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2018-05-20 12:41 GMT
Find Nodes?
    Voting Booth?