in reply to
Parallel Processing, Queueing and Scheduling
Not really perl related, but we've had great
success with openpbs, aka
"Open Portable Batch Scheduler".
- open source
- provides a tcl ( I know, I know )
interface where you can even write your own scheduler
- stdout/stderr watching, as you mentioned
- an open api for reporting status back
- dependency chain ( only run job b if job a succeeds,
I am only using it for running sql batch jobs, as
I needed the job dependency chains. Would have
gotten by with cron otherwise. So, since I'm not
using it like you would, I'm not sure if it covers
your other needs. I will note that it expects to
run scripts, and not binaries ( it reads the script
into stdin for later feeding to whatever is on the
shebang line ). You would have to create wrapper scripts
for your render jobs...
Here's the blurb from their website:
The Portable Batch System (PBS) is a flexible batch queueing and workload management system originally developed for NASA. It operates on networked, multi-platform UNIX environments, including heterogeneous clusters of workstations, supercomputers, and massively parallel systems. Development of PBS is provided by Altair Grid Technologies.