Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^7: Converting a parallel-serial shell script

by kubrat (Scribe)
on Sep 29, 2008 at 10:55 UTC ( #714305=note: print w/ replies, xml ) Need Help??


in reply to Re^6: Converting a parallel-serial shell script
in thread Converting a parallel-serial shell script

See, it made you think :). Your threaded solution assigns a file to a thread to process, so what if one of the files is much larger than the sum of the rest of the files. The whole execution time will be determined by that one big file.

Now, where did you wander off thinking of bin packing. It is nice that you posses such knowledge but it seems to be getting in the way a bit. How about treating the whole set of files as a single file and piping line by line to each of the work nodes in turn? Would that not divide the load equally?

Your suggestion to order the files by sizes when using threads is certainly an improvement that has the advantage of keeping the size and complexity of the code low but it is still sub-optimal.

The main reason I stick with my approach is because it sort of forces you to think about the design of the solution and I mean this in more general terms. Still, your solution is elegant as it can get for this particular problem.


Comment on Re^7: Converting a parallel-serial shell script

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://714305]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2014-09-02 00:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (18 votes), past polls