Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Re: Fork multi processes

by salva (Abbot)
on May 04, 2012 at 07:45 UTC ( #968878=note: print w/replies, xml ) Need Help??

in reply to Fork multi processes

use Net::OpenSSH::Parallel:
my $p = Net::OpenSSH::Parallel->new(workers => 5); for my $host (@hosts) { $p->add_host($host); } $p->all(rsync_put => $src, $dst); $p->run;

Replies are listed 'Best First'.
Re^2: Fork multi processes
by saurya1979 (Initiate) on May 04, 2012 at 08:26 UTC
    Thank you Salva. Is it possible to not use any external Perl module and use build-in modules to achieve it?
      The entire point of Perl is that you do not have to. There are no points to be earned for laboriously doing what has already been done (better) by someone else, such that all you need to do to solve your problem is to install something and then write five or ten additional lines in order to do it.
      Yes, sure, you can achieve it, though it would take you a while to do it.
        Yes I know :-) and I am looking for the way to achieve it. So far I coded like below:
        my @childs; my @allfolders => List of all folders for ( my $i=0; $i<@allFolders; $i+=5 ) { my @sourceFolders = @allFolders[$i..$i+4]; foreach my $folders ( @sourceFolders ) { my $pid = fork(); if ( $pid ) { # parent push(@childs, $pid); } elsif ($pid == 0) { EXECUTe RSYNC HERE.... } else { print "Error: Cannot fork process : $! \n",1); } } foreach ( @childs ) { waitpid($_, 0); } }
        The problem here is that it executes rsync for 5 folders at a time and wait until all of them finishes. BUT, I want to start rsync for 6th folder immediately after 1 of 5 rsync is completed.
      In my case, there is just 1 source and 1 destination server for rsync. I want to rsync directories located at different place on source server to the destination server. So I wanted to run multiple rsyncs to make it faster.
        #!/usr/bin/perl use strict; use warnings; use Net::OpenSSH::Parallel 0.12; use Net::OpenSSH::Parallel::Constants qw(OSSH_ON_ERROR_IGNORE); @ARGV == 3 or die <<EOU; Usage: $0 host src_dir dst_dir EOU my ($host, $src, $dst) = @ARGV; opendir my $dh, $src or die "unable to open $src"; my @ls = grep !/^\.{1,2}$/, readdir $dh; close $dh; my %now; sub on_error { my ($pssh, $label, $error) = @_; warn "transfer of $now{$label} by $label failed\n"; OSSH_ON_ERROR_IGNORE; } sub fetch_next { my ($pssh, $label) = @_; if (@ls) { my $file = shift @ls; $now{$label} = $file; warn "$label is copying $file\n"; $pssh->push($label, rsync_put => "$src/$file", "$dst/$file"); $pssh->push($label, sub => \&fetch_next); } } my $p = Net::OpenSSH::Parallel->new(on_error => \&on_error); $p->add_host("worker$_", $host) for 1..5; $p->all(sub => \&fetch_next); $p->run;
        Requires Net::OpenSSH::Parallel 0.12 I have just uploaded to CPAN.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://968878]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2017-12-14 09:04 GMT
Find Nodes?
    Voting Booth?
    What programming language do you hate the most?

    Results (385 votes). Check out past polls.