Re: Taking advantage of multi-processor architecture

[dragonchild]

To expand on what others have said, you need to take a look at what your program is doing. The important thing to figure out is if there are actions the script does that can be done without knowing if other actions have been completed. In other words, are there actions that can be completely independently of each other? Can you divide-and-conquer the problem a-la the binary search algorithm?

A few rules of thumb:

• If you're modifying files in place, this is generally not a good candidate for SMP^*. This is because you might overwrite changes you made in processA with changes made in processB. However, if you can split the files you're modifying in place, modify them separately, then recombine them, then you have a good candidate for SMP.
• If you're loading a datafile into a database, this is generally a good candidate for SMP. Just split the file into N pieces and launch N copies of your loading program where N is generally twice the number of processors you have. So, if you have 4 processors, split the datafile into 8 pieces and launch a copy of your loading program for each piece.
• If you're building a report from some data, this is generally not a good candidate for SMP by a human. (Databases will handle SMP for you.)

*: SMP = Symmetric Multi Processing - using more than one processor at one time.

[mr_mischief]

I realize that the OP's question undoubtably deals with SMP, but I've made it a mission to point out that TIMTOWTDI when it comes to multiple processor machines.

<pedantic>
SMP = Symmetric Multi Processing - a certain way of using more than one processor at a time, usually reserved to systems of eight or fewer processors. Compare NUMA.

NUMA = Non-Uniform Memory Access - a type of multi-processor arrangement in which memory access times differ depending upon which processor is using which part of memory, necessitating that the OS keep track of which parts of memory it uses or assigns to applications. Compare SMP, see The Linux Scalability Effort for some examples.

Update (for sake of a more complete list): There's also Assymmetric Multi Processing, in which certain code must be run on certain processors instead of any process being assigned to any processor. NUMA is generally closer to SMP than AMP.

Some definitions of SMP only address its differences from AMP, saying that SMP means that any process can run on any processor. This means that most NUMA machines are SMP with additional scheduling concerns.

As I've heard SMP defined, it is usually clarified that not only can any processor handle any process, but that the machine also allows each processor the same access to all of the system's main memory. This definition makes SMP and NUMA distinct.
</pedantic>