One approach is to use OpenPBS
a portable batch system. It includes queueing, parallel processing, prioritization, and more. It is open source and works on Unix systems. There is a perl binding PBS::Client
that provides a nice interface:
# Create a client object linked to a server
my $client = PBS::Client->new;
# Describe the job
my $job = PBS::Client::Job->new(
%job_options, # e.g. queue => 'queue_1', mem => '800
cmd => \@commands
# Optionally, re-organize the commands to a number of queues
$job->pack(numQ => $numQ);
# Submit job
Alternatively, a pure perl solution could be built around Schedule::Depend
. Hmm, it was up to v. 2.6, but doesn't seem to be on CPAN any more...
Update: Fixed the OpenPBS link.