http://www.perlmonks.org?node_id=246451

awkmonk has asked for the wisdom of the Perl Monks concerning the following question:

Oh splendid monks, I seek enlightenment.

I have a CGI script needs to spawn off another Perl routine which may run for up to 3 hours. I therefore need to fire up my second script in background. Easy enough I hear you cry - just shove a & on the end of a system call. BUT, it also needs to work in a windows environment as well as AIX.

So the question is - how do I spawn off a process, without waiting for it to finish, in a multi platform environment?

Thanks in advance

Replies are listed 'Best First'.
Re: spawning Perl scripts
by Abigail-II (Bishop) on Mar 28, 2003 at 12:43 UTC 
    $ cat long.pl
#!/usr/bin/perl

use strict;
use warnings;

print localtime () . ": Hello from $0 ($$)!\n";
sleep 30;
print localtime () . ": Goodbye from $0 ($$)!\n";
__END__

$ cat fork.pl 
#!/usr/bin/perl

use strict;
use warnings;


print localtime () . ": Hello from the parent ($$)!\n";

my $pid = fork;

die "Fork failed: $!" unless defined $pid;

unless ($pid) {
    print localtime () . ": Hello from the child ($$)!\n";
    exec "./long.pl";  # Some long running process.
    die "Exec failed: $!\n";
}

print localtime () . ": Goodbye from the parent ($$)!\n";
__END__

$ ./fork.pl
Fri Mar 28 13:39:20 2003: Hello from the parent (458)!
Fri Mar 28 13:39:20 2003: Goodbye from the parent (458)!
Fri Mar 28 13:39:20 2003: Hello from the child (459)!
Fri Mar 28 13:39:20 2003: Hello from ./long.pl (459)!
Fri Mar 28 13:39:50 2003: Goodbye from ./long.pl (459)!

    [download]
[reply]
[d/l]
Re: spawning Perl scripts
by Aragorn (Curate) on Mar 28, 2003 at 12:29 UTC
    The system calls fork and exec are your friends. Recent versions of Perl for Windows can emulate the fork system call. I don't have any experience with Perl on Windows, but maybe you can use ActiveState's documentation as a starting point.

    Arjen

[reply]
[d/l]
[select]
      fork() on Windows is emulated using threads, so I'm not sure this is the best solution in this situation (as the fork and exec idiom will not work).

      One way to start a process asynchronously is to use system() with an initial argument of 1: 

          if ($^O eq "MSWin32") {
        system(1, "cmd_to_run arguments");
    }
    else {
        # do fork and exec, or use system("cmd_to_run arguments &");
    }

      [download]
      The system(1, ...) call works at least on OS/2 too, and is "documented" in perlport.
[reply]
[d/l]

        By George your right!

        I tried fork and exec, but it just waits for the long process using the win32 Apache. system(1,cmd_to_run) works fine as will system(nohup cmd_to_run &) for the unix version.

        I considered using a trigger file and a separate process, but I prefer a self contained set of scripts.

        I'm forever in your debt.

[reply]
[d/l]
[select]
      my $kidpid = 0; my $jobs_to_fork = 3; my $i_am_the_master = 0; my $samps = 0; my $last_no = 10; my $i = 0; for ($i = 1; $i <= $jobs_to_fork; $i++) { $kidpid = fork_me($kidpid); print "$i. kidpid:$kidpid\n"; } if ($kidpid == 0) { $i_am_the_master = 1; } else { $i_am_the_master = 0; } sleep (3); if ($i_am_the_master) { print "I will look after my children\n"; } else { printf ("I am a child %6d running the program.\n", $kidpid); while ($samps < 1000000) { $rnd_no = int(rand($last_no+1)); $samps++; $total = $total+$rnd_no; } printf ("%6d: %d %f\n", $kidpid, $samps, ($total / $samps)); } sub fork_me { my $kpid = $_[0]; if ($kpid == 0) { $kpid = fork(); # the master preserves himself and start another thread } return($kpid); # the child sill just return to go do his job }
[reply]

        Sorry, I am new here but I have played with perl quite a bit. Here is some code I came up with to spawn processes using fork. It works and it seems quite simple... 

        my $kidpid          = 0;
my $jobs_to_fork    = 3;
my $i_am_the_master = 0;

my $samps   = 0;
my $last_no = 10;
my $i       = 0;

for ($i = 1; $i <= $jobs_to_fork; $i++) {
    $kidpid = fork_me($kidpid);
    print "$i. kidpid:$kidpid\n";
}

if ($kidpid == 0) {
    $i_am_the_master = 1;
} else {
    $i_am_the_master = 0;
}
sleep (3);
if ($i_am_the_master) {
    print "I will look after my children\n";
} else {
    printf ("I am a child %6d running the program.\n", $kidpid);
    while ($samps < 1000000) {
    $rnd_no = int(rand($last_no+1));
    $samps++;
    $total = $total+$rnd_no;
    }
    printf ("%6d: %d %f\n", $kidpid, $samps, ($total / $samps));
}

sub fork_me {
    my $kpid = $_[0];
    
    if ($kpid == 0) {
    $kpid = fork(); # the master perserves himself and start another t
+hread
    }
    return($kpid); # the child sill just return to go do his job
}

        [download]
[reply]
[d/l]
Re: spawning Perl scripts
by phydeauxarff (Priest) on Mar 28, 2003 at 13:07 UTC
    Just to add another way to do this, I have used Parallel::Forkmanager for several scripts that needed to run forked processes. One feature I really appreciate is the ability to limit the number of processes started.

    Here is an example from the DOCs for your convienance. 

      use Parallel::ForkManager;

  $pm = new Parallel::ForkManager($MAX_PROCESSES);

  foreach $data (@all_data) {
    # Forks and returns the pid for the child:
    my $pid = $pm->start and next; 

    ... do some work with $data in the child process ...

    $pm->finish; # Terminates the child process
  }
[reply]
Re: spawning Perl scripts
by nite_man (Deacon) on Mar 28, 2003 at 12:36 UTC
[reply]
[d/l]
Re: spawning Perl scripts
by BrowserUk (Patriarch) on Mar 28, 2003 at 21:34 UTC

    As has been alluded to by others above, the AS implementation of the fork function in the Win32 environment is, an emulation, actually using spawning a native Win32 thread (termed a pseudo-process, see the perlfork html/pod that comes with AS distributions). Whilst in many situations this works well enough, it does mean that the 'parent' thread in the process will hang around dormant until the forked pseudo-process (child thread) goes away.

    The simplist way of starting a detached process to run asynchronously from your CGI (or any other perl script) is to use the system function in conjunction with the Win32 CMD native command START. Eg. Issuing the following command from perl script 

    system( 'start d:\perl\bin\perl.exe -V -e"<>"' );

    [download]

    Note:The command I ave used is only by way of demo. Depending upon your set up, you may not need to specify the full path to the executable, or even name the executable in the command at all.

    will cause the system function to return immediately and a new process will be started to run the command supplied after the start command.

    There a two caveats with this. One is that if the command cannot be found, a pop-up will be displayed and the original script will block until this popup is dismissed, at which point system returns a value of 256 and $! will contain "No such file or directory". I haven't discovered any way of defeating this annoying behaviour.

    The second is that assuming the process to be started is a console application (like perl.exe or simiar), then a new console window will be created for the process. This can be overridden using the /B switch, which means that the new process will share the console of the originating process. Provided that the new process doesn't produce any output or this has been redirected somewhere, then it shouldn't interfere with the CGI program in any way.

    This is similar to the behaviour of commands detached with & in the *inix environment and is easily dealt with.

    Your final, and most flexible option in the Win32 environment is to use Win32::Process. The greater flexibility comes at the cost of some extra complexity and a lack of good documentation within the Win32::Process package itself which forces you to seek Win32 platform documentation on the CreateProcess API. In this, Google is your friend:)

    There are various other options to the start command that may or may not be useful to you. type help start at a command line near you for further details:)

    Examine what is said, not who speaks.
    1) When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
    2) The only way of discovering the limits of the possible is to venture a little way past them into the impossible
    3) Any sufficiently advanced technology is indistinguishable from magic.
    Arthur C. Clarke.
[reply]
[d/l]
[select]
Re: spawning Perl scripts
by paulbort (Hermit) on Mar 28, 2003 at 17:33 UTC
    For security reasons, you might want to be very careful building CGI that runs long or background processes. Could an accident-prone or malicious user kick off many concurrent processes, crashing the machine or corrupting the results?

    Instead of a direct fork()/exec() solution, I would recommend using some sort of indirection or queue, like writing a flag to a file, or a record to a database. A second script then checks for the flag and runs the long process. If the process can wait a minute before starting, the script to check could be fired by cron (or Task Scheduler in Win) every minute and just exit if no flag is found.

    You can even have the second process then occasionally write a record back, or create an HTML page with its status. (I guess you could do that either way, but it's still a good idea to be able to keep tabs on a long process.)

    --
    Spring: Forces, Coiled Again!
[reply]
Re: spawning Perl scripts
by abell (Chaplain) on Mar 28, 2003 at 17:29 UTC

    To add a little side-note to the other Venerable Monks' comments, beware that if you fork a perl program executed by mod_perl, you are forking all of the apache process.

    If you use exec, the process is substituted by the new perl interpreter, so it doesn't really matter, but should you be tempted to just do the new script, you have to take into consideration the cost of the apache+mod_perl process, which will be kept alive until the new script exits.

    Cheers

    Antonio

    The stupider the astronaut, the easier it is to win the trip to Vega - A. Tucket
[reply]
[d/l]
[select]

      On a decent OS, fork doesn't actually copy the entire process, it creates a new process table entry, with process id and all the associated information, but the actual memory pages that contain the memory contents of the two processes are shared until one of the processes attempts to change that memory (a procedure known as copy-on-write). So the actual overhead of forking is minimal, and you won't get multiple copies of even a small portion of the entire apache process, unless one of the processes manages to make a change to it's memory in that tiny amount of time between the fork() and exec() calls, and even then you will only get short-lived copies of the memory pages that actually changed, not the entire process.

      We're not surrounded, we're in a target-rich environment!
[reply]
Re: spawning Perl scripts
by awkmonk (Monk) on Mar 28, 2003 at 12:49 UTC

    Ah, yes - that makes sense now. A fork and an exec. I'll give it a whirl.

    Your a good bunch.

    ++ to you all

[reply]
Re: spawning Perl scripts
by tekkie (Beadle) on Mar 28, 2003 at 17:42 UTC
    I think some people reading awkmonk's post are missing an important factor... the secondary process is being spawned from a CGI script, which creates problems using the exec and system methods everyone has described already.

    exec: "The exec function executes a system command and never returns."
    system: "the parent process waits for the child process to complete."

    For Win32:
    The exec method would cause the script to terminate immediately, which means nothing past the exec() line in the original CGI process would be executed.
    The system command would cause the script to hang and wait for the 3 hour process to complete, most likely causing a CGI timeout.

    Neither of these are very viable options for awkmonk's problem because it needs to be multi-platform and on Win32 the originating CGI process would never run to completion... it would either exit at exec() or hang for 3 hours at system(), however... the sub-process would be spawned and continue running

[reply]
      That's why you fork first, an extremely common action to take before an exec.

      Abigail

[reply]
[d/l]
[select]
        True, this will work in Win32 from the command line. However, when attempting to use a fork/exec solution through an IIS server, the parent still hangs and waits for the child to finish.

        The output I get is:
        Fri Mar 28 12:33:52 2003: Hello from the parent (473)!
Fri Mar 28 12:33:52 2003: Hello from the child (-480)!
Fri Mar 28 12:33:52 2003: Goodbye from the parent (473)!

        [download]
        However, there is a thirty second pause between the display of 'Hello from the child' and 'Goodbye from the parent.' The machine thinks the parent process is done (timestamps), but server keeps crunching until the child terminates, only then does it display whatever is past the exec in the parent.

[reply]
[d/l]

      For Windows OS, I don't see a downside to: 

      $cmd = "start perlprog.pl > file.log";
system ($cmd);

      [download]

      This "system" command should return just after the process is started. One downside is that you will get another cmd window pop up.

[reply]
[d/l]
Re: spawning Perl scripts (Win32/Linux sample)
by gmpassos (Priest) on Mar 29, 2003 at 01:55 UTC
    Well, you can use fork() on Linux that will work very well, and for Win32 is just a new thread (fake fork), since the OS Win32 doesn't have the system fork.

    Here is some sample of how to do that: 

    #!/usr/bin/perl

## Init the detached codes:
sub DETACH_INIT {
  my ( $x ) = @_ ;
  $| = 1 ;  ## Flush STDOUT or will lock some threads on Win32.
  $SIG{CHLD} = \&REAPER ; ## Who will lead with childs on real fork.

  ## Run $x times the detached codes:
  my $wpid = 0 ;
  for my $i (1..$x) {
    next if $wpid ; ## Next until the wait() is on.
    &SPAWN($i) ;
  }

  ## Wait for the childs.
  sub REAPER{
    $wpid = wait ;
    $SIG{CHLD} = \&REAPER ;
  }

  ## Make the fork, start the detached code, than continue main code.
  ## When the detached code return, make an exit of the forked process
+ or thread.
  sub SPAWN {
    my ( $id ) = @_ ;
    my $pid ; 
    if (!defined($pid = fork)) { print "cannot fork: $!\n" ; return ;}
    elsif ($pid) { return ;}
    exit &DETACHED_CODE($id) ;
  }
}

## Code to be runned detached:
sub DETACHED_CODE {
  my ( $id ) = @_ ;
  
  for(0..10) {
    print "$id>> $_\n" ;  
    sleep(1);
  }
}

  ## Tell to start 2 detached codes:
  &DETACH_INIT(2) ;

  ## Some main code to continue:
  for(0..10) {
    print "main>> $_\n" ;
    sleep(1);
  }

    [download]
    But note that is very recomended to use threads directly, and make your program based on thread architecture. But you will need to work on Perl-5.8.0, where thread works fine. With threads you have more resources, specially the shared variables!

    Note, use Perl-5.6.1++ for fork(), specially on Win32.

    Update (execution of other process):
    To execute other scripts from Perl, use open: 

    my $cmd = "$^X foo.pl arg1" ; ## $^X is the path to perl(.exe) interpr
+eter.
open (CMDLOG,"| $cmd") ; close (CMDLOG) ;

    [download]
    If you skip the close you don't wait the process to return: 
    open (CMDLOG,"| $cmd") ;
## Now you can continue your main (caller) script...

    [download]

    Or just use the Open3 from IPC to get the retuned data to the pipe: 

    use IPC::Open3 ;
my $cmd = "$^X foo.pl arg1" ;
open3(LOGREAD , LOGERROR, LOGWRITE, "$cmd");
my @log = (<LOGREAD> , <LOGERROR>) ;
close(LOGREAD , LOGERROR, LOGWRITE) ;
print "@log\n" ;

    [download]
    This 2 ways are very portable, at least between Win32 and Linux. The first (open) is more.

    Other thing, try to use different HANDLES (like CMDLOG) for multi calls when you don't use the close(). For example: 

    open (CMDLOG1,"| $cmd1") ;
open (CMDLOG2,"| $cmd2") ;
open (CMDLOG3,"| $cmd3") ;

    [download]

    Graciliano M. P.
    "The creativity is the expression of the liberty".

[reply]
[d/l]
[select]
Re: spawning Perl scripts
by aquarium (Curate) on Mar 30, 2003 at 12:54 UTC
    You can also write a separate daemon, the daemon being for the long taking process. The daemon just sits and waits for a signal (ipc, semaphore, file, socket, etc.) from the CGI. The daemon can be written to properly restart and control its' use of resources. The daemon can then either fork & exec the 3 hour process and wait for more signals OR it can wait to finish before another 3 hour process is allowed to run. Chris
[reply]
Re: spawning Perl scripts
by dzaebst (Initiate) on May 13, 2008 at 20:10 UTC
    Tried to do this a while ago and had the same fork/exec problem. It's possible to fork/exec and have the page load. Apache waits for the child process to close off its handles before loading. So you just have to close them manually: 
      if (!defined( my $child_pid = fork()))
  { die "Cannot Fork: $!" }
  elsif ($child_pid)
  {
    print "</body></html>";
    exit(0);
  }
  else
  {
    close(STDOUT);  
    close(STDERR);
    close(STDIN);

    exec("$my_command");
    exit(0);
  }

    [download]
    If you're planning on launching processes and don't really need a web interface, it's worth considering perl Tk. OK user interface, fewer security issues, and no quirks introduced by apache.
[reply]
[d/l]

Back to Seekers of Perl Wisdom