Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Re^2: what do you use for job queuing?

by perrin (Chancellor)
on Jun 04, 2004 at 16:15 UTC ( #360971=note: print w/replies, xml ) Need Help??

in reply to Re: what do you use for job queuing?
in thread what do you use for job queuing?

I did think about that, and still might try building one as an example app, but there are some tricky implementation details. For one thing, this could result in a whole lot of programs polling the database frequently. That's not going to scale all that well. We can reduce this by having a dispatcher process on each machine so that there's only one poller per machine, but then it has to handle talking to a bunch of child processes to give them jobs and get their results. (This is trickier than a typical server situation where the children just wait for the next connection to come in on some socket.) That can be done with things like IPC::Run or maybe POE, but it's not trivial.

If I end up trying to build one, I will probably post some design ideas here for feedback.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://360971]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (3)
As of 2020-07-09 02:36 GMT
Find Nodes?
    Voting Booth?

    No recent polls found