Re^2: How to debug perl code that uses threads

[chrestomanci]

On further investigation, (mostly from reading the badly documented code), I discovered the flow and pattern of the program.

• The parent thread is only there to parse command line args, create shared variables etc.
• One thread scans the filing system looking for jobs to do. The job objects are placed into a shared queue.
• Then a variable number of worker threads (up to 30 of them) pop items from the queue of jobs and do the work.
• There was a slight complexity where the workers signal to the work finding thread via a semaphore, so that once the queue has enough jobs in it, the work finding thread pauses until the workers signal that they are idle.

Once I figured out what was going on, it was easier to debug, I simply hacked the code a bit so that queue was populated with a small number of jobs, and then a single worker thread was started.

[BrowserUk]

There was a slight complexity where the workers signal to the work finding thread via a semaphore, so that once the queue has enough jobs in it, the work finding thread pauses until the workers signal that they are idle.

That wiffs a lot of code smell.

Without being party to the actual implementation that allows 30 workers to signal their idleness (or lack thereof) to the finder thread, it generally means that the workers are using dequeue_nb() rather that dequeue().

• And that means that they are effectively polling the queue in busy loops rather than just waiting for something to arrive.

  Even if you regulate that busy loop with a sleep, is still means that the thread has to wake up every so often to see if there is anything to do, whereas if it used the the blocking version, it will get woken up when there is something to do.

  Contrast waking up once every minute to see if it is time to go to work, versus using an alarm clock.

• It can also mean that when the finder thread actually has work to do, it has to compete with 30 workers all taking time slices to see if there's anything in the queue yet.

  The workers may not use many cycles between sleeps in order to check the queue, but in order to wake up at all, the OS has had to do the 100s of thousands of cycles involved in a context switch.

  Think of it like a telephone receptionist trying to conduct a conversation whilst also responding to 30 wrong numbers. The wrong numbers don't take long to deal with, but their affect on the conversation is disastrous.

• Using dequeue_nb() usually requires an additional mechanism be invented for telling the workers there is no more work.

  This because you can no longer use the simple and intuitive $Q->enqueue( (undef) x $nThreads ) to signal the workers they are done.

• Finally, depending how it has actually been implemented, it can lead to feast-famine cycles. (Also known as boom-bust.)
  1. The workers all poll away at the queue waiting for work, meanwhile the finder goes looking for and posts work items.
  2. Then the worker are all busy, so the finder sits doing nothing but polling the semaphore waiting for the workers to run out of work.
  3. goto a.

The better mechanism for controlling the queue size is to have the finder sleep for a short period when the queue size reaches some high water mark. The benefits of this are:

• Only one thread -- the finder -- is ever polling.

  Workers only wake up when there is something for them to do.

• The finder polls on the number of items left in the queue. Ie. sleep 1 while $Q->pending > 30.

  Only one ITC mechanism -- the queue -- is required, which avoid potential conflicts and race conditions; avoids the workers having to do anything to signal the finder.

• The low-water mark level can be easily tailored to suit different hardware/software environments.

  With a little more programming, it can even be dynamically adjusted.

• The finder detects when the queue is running low, before the workers become idle.

  Hence, it can overlap its work with that of the workers and the workers stay busy avoiding the famine scenario.