Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: what do you use for job queuing?

by Itatsumaki (Friar)
on Jun 04, 2004 at 16:03 UTC ( #360964=note: print w/replies, xml ) Need Help??


in reply to what do you use for job queuing?

If I understand the topic correctly, you might want to consider the "trivial" solution of setting up a dispatch table in a database and running monitoring scripts on each machine in the cluster. Each monitor-script would check for new work to process in the database, and update the dispatch table with "in-progress" and "completed" flags as need be. Of course this only works when the individual pieces can easily be broken apart, but I think this is often true for web-applications.

-Tats

Replies are listed 'Best First'.
Re^2: what do you use for job queuing?
by perrin (Chancellor) on Jun 04, 2004 at 16:15 UTC
    I did think about that, and still might try building one as an example app, but there are some tricky implementation details. For one thing, this could result in a whole lot of programs polling the database frequently. That's not going to scale all that well. We can reduce this by having a dispatcher process on each machine so that there's only one poller per machine, but then it has to handle talking to a bunch of child processes to give them jobs and get their results. (This is trickier than a typical server situation where the children just wait for the next connection to come in on some socket.) That can be done with things like IPC::Run or maybe POE, but it's not trivial.

    If I end up trying to build one, I will probably post some design ideas here for feedback.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://360964]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (8)
As of 2020-06-03 10:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?



    Results (22 votes). Check out past polls.

    Notices?