Well, I almost did do this with threads... but ran into some issues. The primary issue is that some of the code called from do_childish_things() is not thread-safe.
in reply to Re: Keeping children alive persistiently, intelligently
in thread Keeping children alive persistiently, intelligently
Another issue (though I'm not *certain* it's really an issue) would have been with memory usage. From what I understand, each thread is a copy of the current interpreter's state, in the same process space as the original invocation. Due to the amount of code that makes up this app, that would be one heck of a big process.
Now, a fork() is supposed to be the same sort of thing, except that the spawned child has it's own address space... but due to the efficiency of fork() on most *nix platforms, it can use copy-on-write against the parent and thus is much more memory efficient. (Indeed, this seems to be the case from my limited inspection of the current running version)
Anyhow - there is no shared state or communication between the processes, except that they all write their output to the same file-handle (opened and initialized by the parent), using flock() to avoid stepping on each other's toes. However, all children are running the same code, and the amount of data resident in any particular child is only 10-20KB *at most*. This seems to work *very* well with fork.
I do appreciate the advice though, but I'm not convinced that using threads would solve any of the requirements I need to fill any easier.