Forking multiple processes at once, but limiting how many at a time

[jLinux]

Im writing a script that will execute a command on a list of servers, and was planning on using fork to do so, however I cant have it go to every server at once, so I was going to have it go to $max amount of child processes, then wait for those to finish. It basically doesnt seem to halt at $max, it just keeps going and forks off all at once. Heres my broken code..

my $max = 5;

while (my ($ip, $hostname) = each(%{$hosts->{'all'}})){
        $i++;
        $c++;

        forkHost($ip);

        if($c == $max) {
                $c = 0;
                1 while (wait() != -1);
        }
}


sub forkHost {
        my $target = shift;
        my $pid;
        next if $pid = fork;    # Parent goes to next server.
        die "fork failed: $!" unless defined $pid;

        if (not $pid) {
                open COMMAND, "/execute/this/command |";
        
                # Do some magic..
        
                close COMMAND;
        
                exit; 
        }
}

Thanks for any help!

[salva]

Both Parallel::ForkManager and Proc::Queue allow one to limit the number of forked processes running concurrently.

If you are using SSH to send jobs to the remote hosts, consider also Net::OpenSSH::Parallel.

[trippledubs]

use strict;
use warnings;
use feature 'say';
use POSIX ":sys_wait_h";
use Time::HiRes qw(usleep);

my $maxProcs = 20; # Maximum number of processes to run concurrently
my $countProcs = 0; # Keep track of how many are running
my $finished = 0; # Flag variable
my $startTime = time(); # Going to keep forking until now + 30

sub main {
        while ($countProcs < $maxProcs) {
                my $child = fork();
                die "Cannot fork: $!" unless defined($child);
                if ($child == 0) {  # Child process
                        doStuff();
                } else {
                        $countProcs++;
                }
        }
        usleep(50); # The main process sleeps so not burning up cpu cy
+cles
                    # in a wait loop
}

sub doStuff {
        my $sleepTime = rand() * 10+3; #3-13 seconds
        printf("%i doing work for %.02f seconds\n",$$,$sleepTime);
        sleep $sleepTime;
        exit;
}

while (not $finished) {
        main();
        #https://www.freebsd.org/cgi/man.cgi?query=waitpid&manpath=SuS
+E+Linux/i386+11.3
        my $reaped =
                waitpid(-1,WNOHANG);  # Wait for any child process wit
+hout blocking
        if ($reaped) {
                say "$reaped reaped";
                $countProcs--;
        }
        $finished = 1 if (time() > $startTime + 30);
}

while (wait() != -1) {
        usleep(50);
}

Thanks Tye for providing the freebsd link which is great information for wait() vs waitpid and which flags to use. I guess that made more sense to me than the perldocs even though I see now I used waitpid exactly as it is in waitpid. Well in my defense it says "do a non-blocking wait for all pending zombie processes" but I guess that just applies to the way it is used in that code snippet.

I wanted "do a non-blocking wait for any pending zombie processes". The code example in waitpid seems to do the same thing as while (wait()!= -1) {}. Hopefully I used it correctly! Maybe I really should get that i-7 for christmas

[builat]

And one more.
It is not good idea to use quotes in Hash or hashref.
Coz $hosts->{'all'} == $hosts={ 'all' => 'foo.bar'};

When you use => operator it already qouted *all* key. So your code is look like $hosts->{'\'all\''};
Most times it works right. But sometimes it could be hard catched error.

[Athanasius]

Hello builat,

I upvoted this, because it raises an interesting issue, but I think your conclusion is incorrect. Consider:

16:36 >perl -wE "my %h = (Fred => 'Wilma'); my ($k, $v) = each %h; say
+ length $k;"
4

16:36 >perl -wE "my %h = (qq['Fred'] => 'Wilma'); my ($k, $v) = each %
+h; say length $k;"
6

16:36 >perl -wE "my %h = ('Fred' => 'Wilma'); my ($k, $v) = each %h; s
+ay length $k;"
4

16:37 >

According to the documentation:

The => operator is a synonym for the comma except that it causes a word on its left to be interpreted as a string if it begins with a letter or underscore and is composed only of letters, digits and underscores....

Otherwise, the => operator behaves exactly as the comma operator or list argument separator, according to context.

Since the string 'all' contains single quote characters, it does not satisfy the criterion “begins with a letter or underscore and is composed only of letters, digits and underscores.” So, it is not subject to the fat comma’s “special quoting behaviour”, but is treated as a normal string expression; that is, the quotation characters function as operators (like q//) and do not make it into the final hash key string.

Hope that helps,

[builat]

I have try it minute ago. You absolutely right. Thank you!