http://www.perlmonks.org?node_id=1081678


in reply to [Thread::Queue] How to handle unexpected termination and time out in threads

First of all, and most importantly, I would challenge your notion of using “thousands of threads.”   It won’t help, and it will hurt, because the processing that you describe is I/O-bound.   Don’t make that disk-drive move its read/write head around any more than you have to, because that is what costs you milliseconds apiece.   Most of the time, in situations that have been multithreaded but are not good candidates for it, all-but-one of your threads will be waiting for that disk-drive.   Remember that multithreading divides the CPU resource, and does not multiply anything.

Secondly ... consider using the xargs command with the -p numprocs option to conveniently spawn numprocs instances of a simple single-threaded program which takes a filename as its command-line parameter and processes just that one file, then dies.   No funky programming required, and you can easily tweak the number of processes.   (You will find the “sweet spot” to be quite small, and it just might be “1.”)   Build a file containing a list of all the filenames to be processed.   (The ls command can do that.)

For multithreaded processing in-general, each thread or process should be the true master of its own affairs, capable of trapping runtime errors and of using a timer-signal to interrupt what it is doing without losing control of the situation.   (Ditto if it is just a single-thread.)   Generally it is not a good idea to put a pistol to the head of any process because you do not know exactly what it was doing when its head exploded.   The process or thread should survive, no matter what, and exit graciously and cleanly.   The parent’s one and only responsibility is to mind the children (aye, ’twas ever thus ...), who do all the work as well as all the communication with the outside world.

Finally, search CPAN to find the many existing frameworks that are already out there to do this very common thing.   (xargs -p being the simplest example.)   Don’t re-invent something that has already been well done.