RE: Threads vs Forking (Java vs Perl)

[JanneVee]

Interesting points. And to contribute to this discussion from a technological point of view. There is a small thing of Multiprocessor vs. Cluster.

I've done a few experiment runs with Mosix and MPI/PVM type of stuff. For those who don't know what I'm talking about. An overview:

Mosix - An augmentation to the Linux kernel to migrate processes from node to node.

MPI - An API for programming multiprocessor/cluster Architectures.
PVM - Almost strictly cluster architectures.

Now as I seen it. I think that mosix is the best tool a "perl"-programmer can use to do cluster stuff. Using simple fork's and if a script runs up processor usage it gets migrated. But I wouldn't call it exactly totaly efficient.

And the relation with MPI/PVM stuff to threading is close. It is a "threading" api that is designed to conserve bandwith.(the first version of MPI didn't have any support for shared variables that I could see)

One common thing among all these things is that communication between processes/threads isn't encouraged by the "standards". To conserve bandwith. So accessing global variables through different threads isn't exactly a good idea for more than one reason...

I wont rant anymore...

/JanneVee

[tilly]

Actually Cluster may have more to do with Multiprocessor than you think.

Larry McVoy has some interesting thoughts on this. Basically you never worry about SMP beyond about 4 CPUs or so. Instead you turn a thousand processor box into a virtual cluster...

Slides for his talk may be found here.

[JanneVee]

Actually Cluster may have more to do with Multiprocessor than you think.

This I know... :)

You can buy a 4 processor SMP or a 16 processor cluster at roughly the same cost. So start worrying. But the point is... It is not a good thing to use global variables (shared variables) for the locking problems(including the fine grained stuff that is in the slides) or bandwith reasons.

But my conclusion on the slides (that I can make out with in a few minutes)... That it is more pro-forking than pro-threading. Am I correct?

/Janne

[tilly]

Well it is more anti-SMP than anything else. :-)

But at a deeper level it is about why forking entire copies of the OS into a virtual cluster should scale better than trying the multi-threaded SMP approach. So yes, it does support forking.

Besides which, if you take this approach, then on monster boxes forking should scale better than threading.

An incidental note. There are problems that the 4 CPU box can outperform the 16-processor cluster at. But for most people the cluster is a better answer, and Linus is very aware of that.