Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Re: Parallel Processing, Queueing and Scheduling

by Anonymous Monk
on Mar 12, 2003 at 00:53 UTC ( [id://242225]=note: print w/replies, xml ) Need Help??


in reply to Re: Parallel Processing, Queueing and Scheduling
in thread Parallel Processing, Queueing and Scheduling

If the jobs are significant pieces of time, the trick that I have used for this is to store information about what jobs are needed in a database. Then let each machine open up a database connection, open a transaction, figure out which job to do, and then mark it as started. It should issue regular updates if desired. Then when it finishes it marks the job as done.

Users have a tool that allows them to add jobs to the database.

I didn't develop this into anything complex, but it wasn't hard to get to a usable state. And since coordination is handled in a lightly loaded database, this should scale to a very large number of machines. And it can coordinate processes that need cross-platform resources. Including human intervention!

Other solutions worth considering are standard clustering technologies like http://www.mosix.org/, and various solutions that fall under the name grid computing.

  • Comment on Re: Re: Parallel Processing, Queueing and Scheduling

Replies are listed 'Best First'.
Re: Re: Re: Parallel Processing, Queueing and Scheduling
by maksl (Pilgrim) on Mar 12, 2003 at 22:02 UTC
    Openmosix contains several improvements by Moshe Bar on the original work of Prof. Barak (it starteted as fork to continue this open project as gpl) try it if you want to use a mosix type linux kernelpatch for clustering :)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://242225]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (9)
As of 2024-07-16 12:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.