moritz has asked for the wisdom of the Perl Monks concerning the following question:
For a $work project we have to define some workflows where individual pieces (henceforth "tasks") run distributed over several machines, and now the big question is: how do we coordinate them?
A typical use case is to run task A, and when it's finished (and successful), run tasks B and C in parallel, and when both are done (and successful), run task D.
The workers will communicate over AMQP (think RabbitMQ).
But we need a piece of software that controls the flow of all theses tasks, and of course I'd like to write in Perl. What existing software could help with that? I think I want some kind of task orchestrator, like a state machine where you can define transitions, forks and joins.
On the task scheduling side, so far I've found Minion. It looks promising, but is very light on high-level documentation. Also while it seems to support events on failed and finished jobs, it doesn't offer any further help with the orchestration. Also no AMQP support, but then I didn't find any perl-based task queues/schedulers that use AMQP.
The state machine side looks pretty bleak. Machine::State and State::Machine both allow only one state, and no joins/forks.
Can you recommend any modules or tools that will help me with coordinating those tasks?
Update: It seems like I'm looking for something like TaskFlow, only in Perl.
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Task orchestrator or distributed state machine
by kschwab (Vicar) on Jul 09, 2014 at 14:01 UTC | |
by kschwab (Vicar) on Jul 09, 2014 at 14:09 UTC | |
Re: Task orchestrator or distributed state machine
by salva (Canon) on Jul 09, 2014 at 13:37 UTC | |
Re: Task orchestrator or distributed state machine
by perlfan (Vicar) on Jul 09, 2014 at 14:12 UTC | |
Re: Task orchestrator or distributed state machine
by salva (Canon) on Jul 09, 2014 at 13:45 UTC | |
Re: Task orchestrator or distributed state machine
by DMR (Novice) on Jul 10, 2014 at 11:37 UTC | |
Re: Task orchestrator or distributed state machine
by choroba (Cardinal) on Jul 10, 2014 at 17:05 UTC | |
Re: Task orchestrator or distributed state machine
by moritz (Cardinal) on Jul 17, 2014 at 14:30 UTC |