|laziness, impatience, and hubris|
Looking for a resource management / job queue moduleby elTriberium (Friar)
|on Jul 20, 2012 at 20:55 UTC||Need Help??|
elTriberium has asked for the
wisdom of the Perl Monks concerning the following question:
This question is not completely Perl-focused as I would be fine with a non-Perl solution, too. However, since our whole setup is currently based on Perl a Perl-based solution would be ideal.
We currently have an environment where we run automated test cases on a combination of Linux servers (S), Linux clients (L) and Windows clients (W). Each individual test might require 0, 1 or more of S, L and W. We have to guarantee that no other tests will run on the same systems at the same time
Currently we have pre-allocated testbeds for several test runs (nightly regression tests and such), but that leaves these systems idle for a long time (a nightly run typically takes ~12 hours, so the systems are idle for the other 12 hours every day). Also, it's making it hard for users to just quickly run some tests as they need a dedicated testbed.
So what we're looking for is a system that can manage these (S, L, W) resources and reserve them for the individual test runs. It should also support a job queue so that a job will be queued when the required resources are currently not available
Once the resources are available it should "reserve" them (or just mark them as reserved in a database) and provide them to our in-house Perl-based tool that launches the tests on them (our tool connects via SSH and Telnet, so that part doesn't need to be included). Once our tool finishes the system I'm looking for should mark the resources as "unreserved" and put them back into the resource pool.
We already looked at many existing solutions, but most don't look completely sufficient:
My question is if something like this is already available? Is anybody doing something similar? I don't expect it to be such an uncommon environment where we have multiple systems and need to reserve them to run tests on them. If there are multiple modules that I can combine (e.g. TheSchwartz and some resource manager) then that would also be fine. I'd appreciate any help!