Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: Looking for a POE based distributed job scheduler

by suaveant (Parson)
on Dec 28, 2009 at 20:48 UTC ( #814662=note: print w/ replies, xml ) Need Help??


in reply to Looking for a POE based distributed job scheduler

I don't know of anything like that but a while ago I was playing with POE a bit on a resource based job scheduler that might at least get you started in the right direction. You are welcome to it if you want. It was pretty much a proof of concept so don't expect great great things, but it looked promising.

                - Ant
                - Some of my best work - (1 2 3)


Comment on Re: Looking for a POE based distributed job scheduler
Re^2: Looking for a POE based distributed job scheduler
by jfroebe (Vicar) on Dec 28, 2009 at 21:44 UTC

    thanks! :) Looks like that might help for part of it but... the more I think about this the more I want the p2p job scheduler.

    One idea that looks like it might have hope is:

    • A message bus (possibly POE::Component::MessageQueue) that will allow job queuing, job statuses, etc to be communicated to the N nodes
    • Static job list, statuses, etc via a replicated db. Possibly using Net::CouchDB - this would also be used as the backend to provide statuses to a front end of some sort (web or similar).
    • Configuration of the peer job 'servers' via a simple ini file (Config::Simple)

    Thoughts on this hair brained idea?

    Jason L. Froebe

    Blog, Tech Blog

      My thoughts would be that you'd probably save yourself a lot of effort by setting up ssh keys on each system and doing all the work on the main site. That way you don't have to do nearly as much synchronization and maintaining of multiple running instances that all need to communicate. Just use remote ps calls, remote run your apps (which makes them local instances sort of) and check /proc/loadavg for system load info. Don't make things harder than you have to. Then the only actual hard part you need to write is the part that decides when to use which computer and how to decide your resource allocation. (my code also handles time based allocation i.e. don't run until at least 4pm)

                      - Ant
                      - Some of my best work - (1 2 3)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://814662]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (14)
As of 2014-07-30 18:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (239 votes), past polls