http://www.perlmonks.org?node_id=711429

katharnakh has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

Background:
I am working on a replication project, where i have to replicate dependencies of projects. I use rsync command to replicate dependencies. Since the dependency list of a project varies and will usually have a huge list, hence i want to it replicate parallely by giving a bunch of file-names to replicate to each rsync command(using --files-from=<filename>) through a file. I am doing the same using perl's threads module.

Problem:
When i have more than one project(a common case), i create 10 threads to run rsync commands. This way i want to achieve replication of all dependencies parallely. I wait in main thread to finish all child threads using join() function of threads module. The script was running fine for somedays, but recently the script terminates, saying 'Segmentation fault', no line number is printed.

I have tried to debug(using perl -d <script-name>) the script, but the debugger does not comeup or no response when it lands on statement to create thread(threads->create(...)), i had to wait long time but i did not comeup so i have to kill the process.

I cannot post the whole code, because it is big and uses some of private packages, but here is how i create 10 threads for each project.

The script terminates by printing Segmentation fault, after executing last print statement in _replicate function. I not able to make out where and why that error is occuring. Or is there a other way to achieve parallel processing withoug using threads? Could someone help me here please?

I am using perl, v5.8.3 built for x86_64-linux-thread-multi

sub _replicate{ my $ref = shift; my $logger = get_logger(); print "Starting replication of dependency files", $/; foreach my $sc(@{$ref}){ next unless (defined $sc); mkdir($LOG_FOLDER."/".$sc->{sc_name}); print "\tScenario: ".$sc->{sc_name}, $/; print "\tLatest Dependencies: ".$sc->{total_dep}." of size "._ +get_readable_size($sc->{total_size}), $/; my @thr_arr = (); print "Creating parallel threads", $/; foreach my $robj(@{$sc->{rsync}}){ my $th = threads->create(\&worker, $robj); # i create thre +ads this way push @thr_arr, $th; } #$logger->info("\twaiting for threads to finish its job..."); print "\twaiting for threads to finish its job...", $/; foreach my $t(@thr_arr){ if (defined $t){ my $k = $t->join(); # this is how i wait for all thre +ads to finish } } #map {my $k = $_->join} threads->list; # map{ # my $th = $_; # my $k = $th->join if($th); # just a blind belief +whether this might cause 'Segmentation fault', hence the check. # }@thr_arr; #$logger->info("\tFinished replicating dependencies of ".$sc-> +{sc_name}); print "\tFinished replicating dependencies of ".$sc->{sc_name} +, $/; } } sub worker{ my $robj = shift; my ($rsync, $server, $from, $to) = @{$robj->{elements}}; my $alt_server = $RSYNC_CONN_STR_2; my $rsync_cmd = $rsync.$server.$from.$to; print "Thread-",threads->self->tid," executing ", $rsync_cmd; }

Thanks in advance,
katharnakh.