Re^2: Perl cgi without mod_perl, your experience

[stvn]

If you have CGI programs doing a relatively long jobs (say you have some programs that do custom image manipulation and that process dwarves the time needed for compilation - or you are doing database queries and the time it takes for the queries is far more than starting the Perl program and setting up the database connection), the savings are minimal.

While not a silver bullet by any means, the Apache Cleanup handler can be very nice. It is essentially the very last phase the the Apache request cycle, and is actually after the last of the headers have been sent to the client (after the request is over form the users perspective).

$r->register_cleanup(\&my_long_running_sub);

I (ab)use it to generate very large DB-query intensive PDFs on several sites. The hijacked process itself stores its progress in a database, and marks a flag when the PDF is done. All the while the users page has been auto-refreshing at a reasonable interval (and tying up a second apache child :-P ), and once the PDF is done and the flag has been set, they can download it.

Sure this can get tricky, since the apache process in a sense becomes "headless" for a while, but with proper exception handling and careful use of alarm you can avoid most of the issues that might come up.

I would argue too, that this approach is actually more effieient since you will save the cost of module loading and have the benefits of database connection pools and other mod_perl goodies at your disposal.

[salvix]

And don't forget to mention that using a reverse proxy will keep your server with a minimal number of mod_perl processes running, i.e., setup an Apache on port 80 (with mod_proxy activated) and redirect every *.pl requests (or all /cgi-bin request, for example) to your mod_perl enabled Apache on another port (81, for instance).

You'll be able to handle even more scripts per second because you will keep the mod_perl server busy only to process the request, and the transmission of the output will be a task for your light Apache on port 80, freeing it to handle another script request.
HTH.

[perrin]

That's actually not the best way to do it. Ideally, you would fork so that your processing does not tie up an apache child process at all. This is how we recommend handling long running jobs on the mod_perl list.

[stvn]

From what I know about fork, and mod_perl, I can't understand why that would be the recommened way to handle this (and I am in no way claiming to know everything about fork/mod_perl, so I may be way off here). My understanding is that forking under mod_perl would result in a duplication of the apache child process, complete with mod_perl and all its goodies. Sure that could then be chopped off of the apache parent and set to live on its own by closing file-handles and such, but then I have a big-fat-apache zombie process which eventually will just need to get reaped by the OS's init process.

How can that be better than hijacking an Apache process with the Cleanup phase for a little while? Am I grossly misunderstanding something here?

[perrin]

You can avoid getting a zombie, and copy-on-write means that most of the memory is shared. This is all described in the mod_perl guide here. Now, if these things you do in the cleanup phase are very short, this is overkill, but for long-running jobs it's really the best option. Another good approach is to use a queue, so that you can just dump jobs onto it and move on, and a separate process monitors the queue and picks up jobs that apache adds to it.