Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re: keeping connection alive while spending time building a zip file

by sundialsvc4 (Abbot)
on Aug 11, 2011 at 13:56 UTC ( #919874=note: print w/replies, xml ) Need Help??

in reply to keeping connection alive while spending time building a zip file

This is, for many reasons, an ideal application for a “job scheduling” system ... running separately from the web server.   Let the web-site become a user interface for that system.

(There are, of course, plenty of things to choose from in CPAN and elsewhere.)

Let’s say that one hundred innocent web-site users happen to, at the same instant, try to build zip-files.   Under your present arrangement, completion of the task by the now-hopelessly-swamped computers could take the better part of a day.

Per contra, those requests could instead cause 100 batch-jobs to be generated, and submitted to a system which will process no more than (say...) 10 jobs simultaneously, with a therefore-assured completion time of no more than (say...) 11 minutes per request worst-case.   Our 100 users now see, variously, that “your request should be ready in about 11 minutes” to “about an hour.”   Maybe they get an e-mail when it’s done.   Maybe they have the ability to decide to cancel their request.   And so on.   Meanwhile, the system(s) who are doing the work will not commit themselves to more simultaneous workload than you know they can deliver consistently:   there is a throttle, a governor, and a queue.

The mere fact that a web-server can be used to request or to initiate something, does not mean that it should also be the one to do it.   In fact, this is (IMHO...) almost universally a bad idea, very-common though it be.

  • Comment on Re: keeping connection alive while spending time building a zip file

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://919874]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (5)
As of 2021-05-15 22:42 GMT
Find Nodes?
    Voting Booth?
    Perl 7 will be out ...

    Results (150 votes). Check out past polls.