|Keep It Simple, Stupid|
Task orchestrator or distributed state machineby moritz (Cardinal)
|on Jul 09, 2014 at 13:19 UTC||Need Help??|
moritz has asked for the
wisdom of the Perl Monks concerning the following question:
For a $work project we have to define some workflows where individual pieces (henceforth "tasks") run distributed over several machines, and now the big question is: how do we coordinate them?
A typical use case is to run task A, and when it's finished (and successful), run tasks B and C in parallel, and when both are done (and successful), run task D.
The workers will communicate over AMQP (think RabbitMQ).
But we need a piece of software that controls the flow of all theses tasks, and of course I'd like to write in Perl. What existing software could help with that? I think I want some kind of task orchestrator, like a state machine where you can define transitions, forks and joins.
On the task scheduling side, so far I've found Minion. It looks promising, but is very light on high-level documentation. Also while it seems to support events on failed and finished jobs, it doesn't offer any further help with the orchestration. Also no AMQP support, but then I didn't find any perl-based task queues/schedulers that use AMQP.
Can you recommend any modules or tools that will help me with coordinating those tasks?
Update: It seems like I'm looking for something like TaskFlow, only in Perl.