Another approach that may be useful is to enqueue the work in a database table, and to retrieve the work using ordinary queries (under transactions .. required in this case). While this of course is not as “efficient,” it does have several useful properties to consider:
- Storage size effectively ceases to be a problem.
- The work can be processed by anyone, at any time, even on any system.
- You can maintain a permanent record of the work performed, and keep statistics.
- The processing is naturally checkpointed, and can be restarted.
- There is a down-side of how to block the processes that are waiting for work .. polling is often used.
To answer the immediate question ... threads and processes are executed in some “fair” (more or less round-robin) fashion, on some CPU, anytime they are eligible to execute. They do not have to “yield” or otherwise give up the processor, and any algorithm which “it seems to work okay™” (famous last words) based on such a principle is
wrong suspect. If you need to keep a queue from growing too large (or any other shared resource), both processes must cooperate in doing so ... as BrowserUK's elegant code example does by blocking the sender while the queue is too-large. (I really do think that you should put this little jewel into CPAN anyway, even if it uses an undocumented feature.)
There are, quite frankly, tradeoffs everywhere. If you use multiple queues and block the sender, then theoretically the sender might be blocked waiting to post to one queue while processes on other queues starve. And so on and on. Creating a truly scalable architecture that will “live long and prosper” under arbitrary real-world loads (and multiple CPUs/cores, etc.) is very difficult to achieve, partly because it’s worlds-away from the developer’s own machine setup. Also consider stepping back from the immediate problem and looking at existing workload-management modules such as POE. There’s more than one way to do it,™ and it’s great to find one that has already been done.