Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re^2: what do you use for job queuing?

by perrin (Chancellor)
on Jun 04, 2004 at 16:15 UTC ( #360971=note: print w/replies, xml ) Need Help??

in reply to Re: what do you use for job queuing?
in thread what do you use for job queuing?

I did think about that, and still might try building one as an example app, but there are some tricky implementation details. For one thing, this could result in a whole lot of programs polling the database frequently. That's not going to scale all that well. We can reduce this by having a dispatcher process on each machine so that there's only one poller per machine, but then it has to handle talking to a bunch of child processes to give them jobs and get their results. (This is trickier than a typical server situation where the children just wait for the next connection to come in on some socket.) That can be done with things like IPC::Run or maybe POE, but it's not trivial.

If I end up trying to build one, I will probably post some design ideas here for feedback.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://360971]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2020-10-24 07:33 GMT
Find Nodes?
    Voting Booth?
    My favourite web site is:

    Results (242 votes). Check out past polls.