defunct process are WAY beyond my experienc

[cristj1]

I'm pretty green and novice-like on PERL. I migrated a 10+ year old Perl script from Solaris 10 (!) to RHEL 7.3, Perl (v5.8.8) to (v5.16.3). It mostly runs great but I continue to have recurring problems with defunct processes. I've read tons of articles on threads, join, $SIG{CHLD} = 'IGNORE', etc. I basically understand I'm not reaping child processes correctly but am way over my head on how to correct it. I could take a week or two to bone up but I hope someone will have pity on this poor beggar and help me out.

The script kicks off a bunch of child processes. They do their thing and the main script SEEMs to try to clean up a bit by simply doing a "join" on each child. This worked fine on the old system. Now, it only seems to work OK when the number of children is small--say less than 50--but seems to result in defuncts when the "family" is much bigger than around 50.

Here are what I think are the salient bits of code. Can you help? I THINK I need to do a "wait" or a $SIG{CHLD} = 'IGNORE', but I don't know enough to just try things. Can you help? Here's where the kids get born:

...
use threads;
...
BEGIN {
   $Config{useithreads} or die "Recompile Perl with threads!";
} # end BEGIN
...
   switch($action) {
      case /adminpass(,|$)/ {
         print "Do adminpass ($hostname)\n" if($verbose || $debug);
         $t[++$#t]=threads->new(\&adminpass);
         next;
      } # end case /adminpass/
      case /backup(,|$)/ {
         print "Do backup ($hostname)\n" if($verbose || $debug);
         $t[++$#t]=threads->new(\&backup);
         next;
      } # end case /backup/
      case /buildhosts(,|$)/ {
         print "Do buildhosts ($hostname)\n" if($verbose || $debug);
         $t[++$#t]=threads->new(\&buildhosts);
         next;
      } # end case /buildhosts/
      case /buildhostsdell(,|$)/ {
         print "Do buildhostsdell ($hostname)\n" if($verbose || $debug
+);
         $t[++$#t]=threads->new(\&buildhostsdell);
         next;
      } # end case /buildhostsdell/
etc...

They do their thing in their respective subroutines, then the main thread does the following in efforts to clean up.

# Loop through all the threads
my $thr;
foreach $thr (threads->list) {
   # Don't join the main thread or ourselves
   if ($thr->tid && !threads::equal($thr, threads->self)) {
      $x=$thr->join;
      print 'TID ' . $thr->tid() . " returned: $x\n" if($verbose || $d
+ebug);
   } # end if
} # end foreach

# Exit the script cleanly
exit 0;

I REALLY appreciate your help!

[zentara]

The script kicks off a bunch of child processes.

Hi, I think you are confusing defunct processes with threads. When using threads, the spawned threads can either be joined or detached. If you detach it, it dosn't need to be joined, it will cease when the code block ends. When the parent thread exits, it will take all child threads with it, and issue a warning that some child threads were running at exit.

As usual, you should should make a complete self-contained running example. Here are a few to show the difference between detach and join.:

#!/usr/bin/perl
use strict;
use warnings;
use threads;

foreach (0..100){
  # create a thread to call the "run_system" subroutine
  # to run "ls -la" on the OS
  my $thread = threads->create("run_system", "ls -la");
   $thread->detach;
 }

sub run_system{
    my $cmd = shift;
    system($cmd);
}

__END__
#1liner
perl -Mthreads=async -le"async{ system 'dir'; } for 1 .. 100"

Joining is a bit different.

#!/usr/bin/perl
use warnings;
use strict;
use threads;

# join() does three things: it waits for a thread to exit,
# cleans up after it, and returns any data the thread may
# have produced. 

my $thr1 = threads->new(\&sub1);
my $ReturnData1 = $thr1->join;
print "Thread1 returned @$ReturnData1\n";

my $thr2 = threads->new(\&sub2);
my $ReturnData2 = $thr2->join;
print "Thread2 returned @$ReturnData2\n";

my $thr3 = threads->new(\&sub3);
my $ReturnData3 = $thr3->join;
print "Thread3 returned @$ReturnData3\n";


sub sub1 {
print "In thread1.....\n";
sleep 5;
my @values = ('1a','1b', '1c');
return \@values; 
}

sub sub2 {
print "In thread2.....\n";
sleep 5;
my @values = ('2a','2b', '2c');
return \@values; 
}

sub sub3 {
print "In thread3.....\n";
sleep 5;
my @values = ('3a','3b', '3c');
return \@values; 
}

[huck]

In preparation of trying to suggest a replacement for your switch use i would like to start with this comment

$t[++$#t]=threads->new(\&adminpass); seems to be a real verbose way of saying push @t,threads->new(\&adminpass);

It looks like your use of switch is to process a comma separated list of commands. a simple replacement for

   switch($action) {
      case /adminpass(,|$)/ {
         print "Do adminpass ($hostname)\n" if($verbose || $debug);
         $t[++$#t]=threads->new(\&adminpass);
         next;
      } # end case /adminpass/
      case /backup(,|$)/ {
         print "Do backup ($hostname)\n" if($verbose || $debug);
         $t[++$#t]=threads->new(\&backup);
         next;
      } # end case /backup/
      case /buildhosts(,|$)/ {
         print "Do buildhosts ($hostname)\n" if($verbose || $debug);
         $t[++$#t]=threads->new(\&buildhosts);
         next;
      } # end case /buildhosts/
      case /buildhostsdell(,|$)/ {
         print "Do buildhostsdell ($hostname)\n" if($verbose || $debug
+);
         $t[++$#t]=threads->new(\&buildhostsdell);
         next;
      } # end case /buildhostsdell/
...
}

might be

      if ($action=~/adminpass(,|$)/) {
         print "Do adminpass ($hostname)\n" if($verbose || $debug);
         push @t,threads->new(\&adminpass);
      } # end case /adminpass/
      if ($action=~/backup(,|$)/) {
         print "Do backup ($hostname)\n" if($verbose || $debug);
         push @t,threads->new(\&backup);
      } # end case /backup/
      if ($action=~/buildhosts(,|$)/) {
         print "Do buildhosts ($hostname)\n" if($verbose || $debug);
         push @t,threads->new(\&buildhosts);
      } # end case /buildhosts/
      if ($action=~/buildhostsdell(,|$)/) {
         print "Do buildhostsdell ($hostname)\n" if($verbose || $debug
+);
         push @t,threads->new(\&buildhostsdell);
      } # end case /buildhostsdell/
...

but a more efficient and direct solution may be as follows

 
# create a list of the command names and the subroutines they call 
my %cmds=(adminpass      =>\&adminpass,
          backup         =>\&backup,
          buildhosts     =>\&buildhosts,
          buildhostsdell =>\&buildhostsdell
          ); 
# break list of comma sep commands into words 
#  and if we have a subroutine for that word start a thread with it  
for my $cmd (split(',',$action)) { 
      if (exists $cmds{$cmd}) { 
         print "Do $cmd ($hostname)\n" if($verbose || $debug);
         push @t,threads->new($cmds{$cmd});
      } 
      else { 
        print "Command $cmd unknown\n"; 
      } 
}

[huck]

While i am far from truly knowing much about these things i agree with zentara's statement I think you are confusing defunct processes with threads. I believe defunct process are the result of exec, fork or certain types of system calls. It is those that you have to wait on. I think to solve this problem we will need to see the areas of your code that use those rather than where you start threads.

Someone else more knowledgeable than me will probably also pipe up about the fact that the use of switch is discouraged. While it MAY not be the cause of your current fire you may want to look into ways to replace its use with more "modern" perl code

[karlgoethebier]

Another sketch for further inspiration:

#!/usr/bin/env perl

#     $Id: async.pl,v 1.8 2017/07/03 19:46:43 karl Exp karl $

use strict;
use warnings;
use threads;
use MCE::Hobo;
use feature qw(say);

my %actions = (
    nose => sub {
        my $hobo = mce_async { shift } "kumquat";
        ( $hobo->pid, $hobo->join );
    },
    foo => sub {
        my $hobo = mce_async { shift } "bar";
        ( $hobo->pid, $hobo->join );
    },
);

say join " ", $actions{q(nose)}->();
say join " ", $actions{q(foo)}->();

__END__

Please note: if you say # use threads it forks.

This is not quite correct:

"...Only the shared-server is a thread when loading threads. Hobo workers are spawned via fork even when loading threads." marioroy

See also MCE::Hobo

Best regards, Karl