Perl / Apache 2 / Alarms

[DanielSpaniel]

Versions: Perl 5.10, Apache 2.x, CentOS 6

(modified since originally posted, so that process A kicks off the scripts, rather than using an intermediary script to do it)

I have a Perl script (process A) which kicks off several other Perl scripts. Process A is kicked off via http.

I am trying to get process A to wait for x seconds after having executed the other scripts, which may have finished, or they may not have done; I don’t really care – I just need to wait x seconds in process A, and then continue.

I’ve tried several ways of doing this, but can’t seem to get it working as I want. Partly this is probably because I don’t really understand signals etc, etc properly. I used to have a way of making it work in Apache 1.3, but Apache 2.x has changed how it handles such things I think.

So, code which I can get to work as a single process (purloined from StackOverflow), as a test, is below:

#!/usr/bin/perl

use strict;

use My_Functions;

&print_myheader();

my $br=(-t)?"\n":'<br />';

use POSIX qw(:signal_h);

my $sigset_new = POSIX::SigSet->new();
my $sigset_old = POSIX::SigSet->new();

sigprocmask(SIG_BLOCK, $sigset_new, $sigset_old);

if ($sigset_old->ismember(SIGALRM)) {
  print "SIGALRM is being blocked!$br";
  $sigset_new->addset(SIGALRM);
  sigprocmask(SIG_UNBLOCK, $sigset_new);
} else {
  print "SIGALRM NOT being blocked$br";
}

$SIG{ALRM} = sub {print scalar(localtime()), " ALARM, leaving$br"; sig
+procmask(SIG_BLOCK, $sigset_new, $sigset_old); exit; };

alarm(5);

print scalar(localtime()), " Starting sleep...$br";

sleep (10);

print scalar(localtime()), " Exiting normally...$br";

This does as expected, and exits after five seconds.

However, I’m not sure if it’s because I am not implementing the code properly, but I just can’t seem to get it working, and I can get no http output until each of the kicked-off scripts has finished.

The relevant snippet of my code from process A looks like this:

[…. code …]

my $sigset_new = POSIX::SigSet->new();
my $sigset_old = POSIX::SigSet->new();

sigprocmask(SIG_BLOCK, $sigset_new, $sigset_old);

if ($sigset_old->ismember(SIGALRM))
{
$sigset_new->addset(SIGALRM);
sigprocmask(SIG_UNBLOCK, $sigset_new);
}
else
{
# do nothing
}

eval {

$SIG{ALRM}=sub{
   sigprocmask(SIG_BLOCK, $sigset_new, $sigset_old);
   exit;
};

alarm ($timer);

   for (@mylist)
   {
   my $cmd= "perl process_$_.pl args";  # tried with trailing & too
   system($cmd);
   }

alarm (0);
};

[ … more code …]

Any assistance would be greatly appreciated! Thanks.

[Eliya]

I can get no http output until each of the kicked-off scripts has finished.

The important thing here is to close the standard file handles in the child processes. In particular stdout and stderr — stdin is not an issue here, as the associated pipe is for sending data from Apache to the CGI script, and when done, Apache takes care of closing this pipe itself.

Background:  when Apache runs an external program (CGI script) to generate content, it creates three pipes to that process:

           ---------                  -----------------
          |         |   form data    |                  |
          |         | -------------> | stdin            |
          |         |                |                  |
   -----> |         |   HTML content |           CGI    |
  browser |  Apache | <------------- | stdout    script |
   <----- |         |                |                  |
          |         |   error msgs   |                  |
          |         | <------------- | stderr           |
          |         |                |                  |
           ---------                  -----------------

The first one, which is connected to stdin of the CGI script, is used for sending form data, etc. to the script. The second one, which is connected to stdout of the script, is used for reading any content the script produces. And the last one, connected to the script's stderr, is used to read error messages, if there should be any. (This by default ends up in the webserver's error log, but may also be redirected to the browser.)

When the CGI script creates further child processes via fork or fork/exec (and system also is fork/exec under the hood), all file handles are duplicated due to the fork

           ---------                  -----------------
          |         |   form data    |                  |
          |         | -------------> |    -----------------
          |         | -----------------> |                 |
   -----> |         |   HTML content |   |                 |
  browser |  Apache | <------------- |   |                 |     
   <----- |         | <----------------- |   child         |
          |         |   error msgs   |   |   process       |
          |         | <------------- |   |                 |
          |         | <----------------- |                 |
           ---------                  ---|                 |
                                          -----------------

and Apache will wait for all of them to be closed, before it stops reading from the script's stdout/stderr and consideres the dynamic content generation to be finished (Apache has to wait because it cannot tell when the script has finished generating content — which is normally indicated by the script closing the pipe).  So, unless the child processes do close the handles themselves, this won't happen before the processes terminate.

In short, with long running background processes, you have to close stdout/stderr yourself (or redirect them to a file, if they are being used).  As system provides no direct way to manipulate the duplicated file handles, it's usually easiest to explicitly fork/exec, which gives you more control:

my $pid = fork;
die "fork failed: $!" unless defined($pid);
if ( $pid == 0 ) {
    # child
    close STDOUT;    # <--- !!
    close STDERR;    # <--- !!
    ...
    exec $program, @args;
    die "exec failed";
}

Another thing to consider is that the background processes typically won't automatically terminate when the alarm fires. So if you want them to terminate, just kill them yourself (which is easy, as you have their PIDs).  And in case the background processes do (or may) fork further childrem, it's usually best to create a separate process group for them (setpgrp), and then kill the entire group by sending a negative signal number to the ID of the process group.

Also, in case you should be running in a persistent environment (FCGI, mod_perl), don't forget to wait/waitpid for the terminated child processes, or else zombies will accumulate. (For a regular CGI script, this is not an issue, because as soon as the respective parent (the main CGI script) terminates, any of its zombies will be taken care of by the OS.)

[DanielSpaniel]

Thanks very much for that extremely helpful post Eliya. Much appreciated. I'm not at my machine right now, but will study your comments more closely later on when I am, and make sure I close stdout/stderr. Thanks also for the excellent diagrams!

[DanielSpaniel]

Yes, that's excellent. I now have it working perfectly, thanks to your suggestions, and the code snippet provided. It's doing exactly what I need it to!

Thanks again.

[pileofrogs]

Let me see if I understand correctly:

You have a script that you want to launch some other scripts, then have it wait around a bit and then exit, right?

You need to fork off those other processes. The easy way to do that in unix is to stick a '&' at the end of the command. Using fork() and exec() is also a popular combination.

See perlfork & perlipc.

unix example:

system("ls -l > ls.out");  # you gonna have to wait
system("ls -l > ls.out &"); # the command goes into the
                            # background

fork & exec example

my $pid = fork;
die "Fork failed me!\n" if !defined($pid);
if ( $pid == 0 ) {
   # I'm the child!
   exec("ls -l > ls.out");
}
# I'm the parent!
sleep 5; #or whatever

[Anonymous Monk]

Proc::Background

[DanielSpaniel]

Thanks for the reply ... I'll clarify. After executing the scripts it should wait a maximum of x seconds, or until each script has completed if that is sooner, and then continue processing the original script - and then exit. That's why using alarms is useful.

I thought that exec (as you've suggested) would simply replace the currently executing process, but I'm open to experimenting.

However, I would prefer to get the existing code working if possible, since I know can work (first example). I'm just not certain I am implementing it correctly (second example) in my own code, and would like somebody more Perlish than myself to suggest/help with that if possible.

Appreciate your comments though.