in reply to
More than just a config file
Have you considered just using mon? It's built for this.
I didn't make it all the way through the dox, but would "mon" accept external tester scripts?
Results (211 votes),