|Perl: the Markov chain saw|
Re^3: Keeping browser active while computing in Perl/CGIby sundialsvc4 (Abbot)
|on Jul 30, 2014 at 19:02 UTC||Need Help??|
I intended that a single computation would be a single job ... not a series of multiple jobs.
What I meant to say was that, if the user had several of those computations to run (using different input data-sets), each one would be its own “job.” It’s a self-contained description of what program to run and what data to give it. There would be another process or set of processes, somewhere, on some number of computers (or the same one), which would be dedicated to running them and making the results available by some convenient means. These are the “batch job runners,” or “initiators,” that actually do the work, perhaps on über-fast machines that do nothing else. The web-page would thus become a user-interface to that overall process, allowing computations to be requested (with specified inputs supplied using the page), monitored, and results retrieved. You don’t have to “wait for” anything ... just check-back in a few minutes to see if it’s done yet.
Yes, of course the idea could be extended to support parallelism, clustered computing and so-on, if the nature of the problems being solved were conducive to that sort of thing. I’ve seen some dedicated workflow-tools, homebrewed by national labs and such, that know a lot about a particular type of work that they spend all day solving, and, having been built with public funds, they are now publicly available. There’s actually a lot of “prior art” for doing things like this, as well as various commercial products. It will be a “pick one vs. build” decision that probably will be, “pick one.”
I’ve had great successes with this sort of thing. For one thing, a good batch-monitor can start another unit of work without incurring any idle-time between jobs, even if the jobs take a little more or a little less time. Unlike cron, there will never be any collisions, nor any wasted time, nor any under- or over-utilized resources. And, over the course of a few days or what-not, all that time really adds up. One client easily pulled 30% more work out of his existing equipment, and was quite surprised at the difference. It was just wasted time, no longer being wasted! The power-bills went up somewhat, but they didn’t care. Work that “might or might not” actually be finished by morning ... always was. The ability to do automatic checkpoint/restart was just a bonus.