parallelisation (how to wait on N threads)

[erroneousBollock]

Update: I, like the following commenters, don't think that threads are appropriate in this case, but I'll answer the question in case someone comes looking for an answer where threads would be appropriate.

My problem is that i don't see how i can check if ANY of the 4 workers has finished.

Have a single control queue shared between the worker threads and the main thread.

Create a Thread::Queue object, which we'll call the "control queue". When you create the workers, pass that queue to each of the workers you create. When a worker is done it should enqueue a "worker N done" message into the "control queue".

In your main thread, once all the worker threads are started, do a blocking dequeue from the "control queue" in while a loop, where the loops ends when you've received the appropriate number of "worker N done" messages. (ie: You could receive other useful messages on the same queue).

In the body of the while loop, you can deal with what should happen when ANY of the workers finish (use a condition variable if you're only interested in doing so for the first).

After the loop, you can deal with what should happen when ALL of the workers have finished.

re: threads and system():

Also, you seem to be doing system() (which is fork() and exec()) inside the worker thread. Are you sure you understand the implications of mixing fork() with Perl threads?

-David

[Cefu]

David,

Could you please elaborate on the implications of mixing fork() with Perl threads?

I'd like to know what the pitfalls are. While it's clear that threads are redundant (system()/fork() already allows the processes to run in parallel) Jochen's approach still seems natural to me. Starting four threads from the main proccess allows you to limit how many processes you are running at once and to use four identical workers to handle the input/output to each fortran executable.

If it's best to avoid mixing fork() (from the system() calls) and Perl threads, what would you suggest for Jochen's situation? Using only system() calls to fork from the main process while the main loop maintains a count to be sure no more than four run at once?

Cefu

[erroneousBollock]

Could you please elaborate on the implications of mixing fork() with Perl threads? ... I'd like to know what the pitfalls are.

The implications stem from how Perl threads are implemented.

Threads.pm clones all non-shared data in the 'parent' thread to the 'child' thread. If you have a lot of data (or code) loaded in the spawning thread, that data is cloned (copied) into the spawned thread.

Users of Unix fork() have come to expect Copy-on-Write semantics; only copying data when it is written to. While that expectation is not necessarily warranted for perl's fork() (perl modifies bit flags on its variables routinely), it certainly doesn't work that way for threads... if you don't understand the behaviour, you'll end up using lots more memory than you intended.

What I don't know (I'm sure BrowserUK does) is whether all the running threads of the (forking) process are then present in the newly spawned (child) process; if they are, it could get messy ;) Tested: only active thread is forked().

If it's best to avoid mixing fork() (from the system() calls) and Perl threads, what would you suggest for Jochen's situation?

I'd recommend he use Parallel::ForkManager to manage worker processes... that's exactly what it was written for.

Also, remember that fork() is emulated using Threads on Win32, so that wouldn't necessarily be the best way to go on that platform.

-David