http://www.perlmonks.org?node_id=360971


in reply to Re: what do you use for job queuing?
in thread what do you use for job queuing?

I did think about that, and still might try building one as an example app, but there are some tricky implementation details. For one thing, this could result in a whole lot of programs polling the database frequently. That's not going to scale all that well. We can reduce this by having a dispatcher process on each machine so that there's only one poller per machine, but then it has to handle talking to a bunch of child processes to give them jobs and get their results. (This is trickier than a typical server situation where the children just wait for the next connection to come in on some socket.) That can be done with things like IPC::Run or maybe POE, but it's not trivial.

If I end up trying to build one, I will probably post some design ideas here for feedback.