Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re: Keeping browser active while computing in Perl/CGI

by sundialsvc4 (Abbot)
on Jul 30, 2014 at 15:30 UTC ( #1095624=note: print w/replies, xml ) Need Help??

in reply to Keeping browser active while computing in Perl/CGI

If you have something which takes a digital computer “20 minutes or so” to do, then what you’ve really got is a batch job.   And so, you need to select a batch-processing system to do it.   There are many to choose from.   They aren’t specific to Perl.   Many are free.   Some are very full-featured, handling clusters, checkpoint/restart, and so on, and are open-source.

Or, you could “roll your own,” using cron jobs and so-forth.   A database table can serve as the queue:   the web-site user adds a new entry to this queue and, through the website, can observe the progress of the job and can collect the results when the job ends.   He can go to lunch in the meantime.   Also, the system is arranged such that, no matter how many jobs are submitted, they are always processed at a predictable and sustainable rate.   The user could submit 30 jobs all at once, notice that the system is running them (say) three at a time, and know when to check back to see which ones have completed and which ones are still yet-to-do.   No matter how big the backlog gets, the system continues to crank through them at what has been pre-determined to be the fastest sustainable rate, allowing a predictable level-of-service to be maintained.

  • Comment on Re: Keeping browser active while computing in Perl/CGI

Replies are listed 'Best First'.
Re^2: Keeping browser active while computing in Perl/CGI
by Anonymous Monk on Jul 30, 2014 at 16:23 UTC
    If you have something which takes a digital computer "20 minutes or so" to do, then what you’ve really got is a batch job.

    That's incorrect. A single computation that takes a long time is not the same as a series of multiple jobs.

    OP never specified which it is, or whether the computation has steps that can be executed individually - so that's a question for you, vit?

      I intended that a single computation would be a single job ... not a series of multiple jobs.

      What I meant to say was that, if the user had several of those computations to run (using different input data-sets), each one would be its own “job.”   It’s a self-contained description of what program to run and what data to give it.   There would be another process or set of processes, somewhere, on some number of computers (or the same one), which would be dedicated to running them and making the results available by some convenient means.   These are the “batch job runners,” or “initiators,” that actually do the work, perhaps on über-fast machines that do nothing else.   The web-page would thus become a user-interface to that overall process, allowing computations to be requested (with specified inputs supplied using the page), monitored, and results retrieved.   You don’t have to “wait for” anything ... just check-back in a few minutes to see if it’s done yet.

      Yes, of course the idea could be extended to support parallelism, clustered computing and so-on, if the nature of the problems being solved were conducive to that sort of thing.   I’ve seen some dedicated workflow-tools, homebrewed by national labs and such, that know a lot about a particular type of work that they spend all day solving, and, having been built with public funds, they are now publicly available.   There’s actually a lot of “prior art” for doing things like this, as well as various commercial products.   It will be a “pick one vs. build” decision that probably will be, “pick one.”

      I’ve had great successes with this sort of thing.   For one thing, a good batch-monitor can start another unit of work without incurring any idle-time between jobs, even if the jobs take a little more or a little less time.   Unlike cron, there will never be any collisions, nor any wasted time, nor any under- or over-utilized resources.   And, over the course of a few days or what-not, all that time really adds up.   One client easily pulled 30% more work out of his existing equipment, and was quite surprised at the difference.   It was just wasted time, no longer being wasted!   The power-bills went up somewhat, but they didn’t care.   Work that “might or might not” actually be finished by morning ... always was.   The ability to do automatic checkpoint/restart was just a bonus.

        Until we hear more from vit about the specs, just say to yourself, "gloves".

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1095624]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2018-05-22 01:22 GMT
Find Nodes?
    Voting Booth?