http://www.perlmonks.org?node_id=919874


in reply to keeping connection alive while spending time building a zip file

This is, for many reasons, an ideal application for a “job scheduling” system ... running separately from the web server.   Let the web-site become a user interface for that system.

(There are, of course, plenty of things to choose from in CPAN and elsewhere.)

Let’s say that one hundred innocent web-site users happen to, at the same instant, try to build zip-files.   Under your present arrangement, completion of the task by the now-hopelessly-swamped computers could take the better part of a day.

Per contra, those requests could instead cause 100 batch-jobs to be generated, and submitted to a system which will process no more than (say...) 10 jobs simultaneously, with a therefore-assured completion time of no more than (say...) 11 minutes per request worst-case.   Our 100 users now see, variously, that “your request should be ready in about 11 minutes” to “about an hour.”   Maybe they get an e-mail when it’s done.   Maybe they have the ability to decide to cancel their request.   And so on.   Meanwhile, the system(s) who are doing the work will not commit themselves to more simultaneous workload than you know they can deliver consistently:   there is a throttle, a governor, and a queue.

The mere fact that a web-server can be used to request or to initiate something, does not mean that it should also be the one to do it.   In fact, this is (IMHO...) almost universally a bad idea, very-common though it be.