_replicate() is passed with a datastructure and it looks like this,

$VAR1 = [
          {
            'sc_name' => 'XYZ_Scenario',
            'rsync' => [
                         {
                           'elements' => [
                                           'rsync --archive --relative
+ --stats --verbose --links --copy-links --copy-unsafe-links --safe-li
+nks --times  --files-from=\'./Sep_17_2008(22h.50m.52s)/gen/file-from/
+XYZ_Scenario/rsync_input-1.txt\' ',
                                           'rsync://xxx.yyy.zzz.corp:1
+873/',
                                           'contexts',
                                           ' /var/workshare/contexts >
+> \'./Sep_17_2008(22h.50m.52s)/log/XYZ_Scenario/rsync_input-1.log\' 2
+>&1'
                                         ],
                           'statistics' => {
                                             'total_files' => 863,
                                             'size' => 232563375
                                           },
                           'status' => undef # here i write something 
+meaningful for db updation later.
                         },
                        ... 
                        ...
                        ]# this array will have 10 or less, such objec
+ts with diff. 'rsync_input-*.txt' files to replicate
            'total_size' => 2184209735,
            'total_dep' => 13725
          },
          ... 
          ...
        ]# varies, based on scenarios
[download]

I checked this datastructure carefully, it looks like what i intended, so no problem till here

Below is the same set of functions im posting as earlier. Because i think the problem lies here. I tried running my script on perl5.8.8 and still i get 'Segementation fault' even if i actually execute the rsync command in thread or just print rsync command in thread and return. I strongly believe, i might be calling join() method on thread object which might have died after finishing its job. Hence i try to dereference a reference which is deallocated(may be or ..?).

This happens because, when 10 threads are running parallely and i wait for a 2nd thread, suppose, to join. Meanwhile 3rd or 4th or 8th(anything till 10) might have finished running. Once 2nd joins and main thread tries to call join() on next thread object, in the array(either returned by threads->list or i keep thread object in a array), which no more exists, or no clue whether the thread is joinnable.

I tried to make sure whether thread is running as you can see in below code, _replicate(),

sub _replicate{
    my $ref = shift;
    my $logger = get_logger();
    print "Starting replication of dependency files", $/;
    $logger->info("Starting replication of dependency files");
    foreach my $sc(@{$ref}){
        next unless (defined $sc);
        mkdir($LOG_FOLDER."/".$sc->{sc_name});
        $logger->info("\tScenario: ".$sc->{sc_name});
        $logger->info("\tLatest Dependencies: ".$sc->{total_dep}." of 
+size "._get_readable_size($sc->{total_size}));
        my @thr_arr = ();
        foreach my $robj(@{$sc->{rsync}}){
            # I will add a key to this datastructure, to check whether
+ thread is joinnable or it is still running? 
            $robj->{thr} => 'running';
            
            my $th = threads->create(\&worker, $robj);
            $logger->info("\tThread-".$th->tid.", Total files: ".$robj
+->{statistics}->{total_files}.", Size: "._get_readable_size($robj->{s
+tatistics}->{size})."[".$robj->{statistics}->{size}."B]");
            $logger->info("\tcmd: ".join("", @{$robj->{elements}}));
            push @thr_arr, $th->tid;
        }
        $logger->info("\twaiting for threads to finish its job...");
        
        # 3rd try
        foreach my $k(0..$#thr_arr){            
            # lets check tid and then access the thread object!
            print $k," ",$sc->{rsync}->[$k]->{thr}, $/;
            if ($sc->{rsync}->[$k]->{thr} eq 'running'){# if not, thre
+ad might have died and we try to acces the mem. which is deallocated 
+after thread's death
                my $t =  $thr_arr[$k];
                my $th = threads->object($t);
                $th->join() if ($th);
            }
        }

        # 2nd try
        # map{
            # my $th = $_;
            # just a blind belief whether this might cause 'Segmentati
+on fault', hence the check. But here may, the thread object im referr
+ing might have been deallocated due to death of thread, hence i get '
+Segmentation fault' .... ?
            # my $k = $th->join if($th);        
        # }@thr_arr;
        
        # 1st try
        # May be the thread objects returned by threads->list are unjo
+ined, but are they joinnable? no clue...!
        #map {my $k = $_->join} threads->list;
        
        $logger->info("\tFinished replicating dependencies of ".$sc->{
+sc_name});
    }
}

sub worker{
    my $robj = shift;
    my ($rsync, $server, $from, $to) = @{$robj->{elements}};
    my $alt_server = $RSYNC_CONN_STR_2;

    print "Thread-".threads->self->tid." running";
    my $i = 0;
    while(++$i <= $MAX_REPL_ATTEMPT){
        #$logger->info("\t\t[Attempt-".$i."]Thread-".threads->self->ti
+d." executing [".$rsync_cmd."]");
        #$logger->info("\t\t\tTotal files: ".$robj->{statistics}->{tot
+al_files}.", Size: "._get_readable_size($robj->{statistics}->{size}).
+"[".$robj->{statistics}->{size}."B]");
        my $rsync_cmd = $rsync.$server.$from.$to;
        `$rsync_cmd`;
        if ($?){ # because of connection refusal from server, command 
+fails
            $robj->{status} = "Completed with error!";
            $rsync_cmd = $rsync.$server.$from.$to;
            $server = ($i%2) ? $RSYNC_CONN_STR_1 : $RSYNC_CONN_STR_2; 
+# just a small trick to use other port on the same server for connect
+ion
            #$logger->error("ERROR: Thread-".threads->self->tid." says
+, replication Attempt-".$i." failed, trying again after 2 mins.");
            sleep(120);
        }else{
            $robj->{status} = "Completed";
            last;
        }
    }
    $robj->{thr} = 'done';
    my $etime = time;
    
    my $spent_time = $etime - $stime;
    my $logger = get_logger();
    $logger->info("\t\t[Attempt-".$i."]Thread-".threads->self->tid." t
+ook "._format_spent_time($spent_time)." time");
}
[download]

I would ask, is there anyway i would make sure all threads are finished or call join on only those threads which are joinnable or i have to go with other solution which sent earlier, fork() ing processes, instead thread?

Thanks in advance,
katharnakh.