Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: Fork multi processes

by salva (Abbot)
on May 04, 2012 at 07:45 UTC ( #968878=note: print w/replies, xml ) Need Help??

in reply to Fork multi processes

use Net::OpenSSH::Parallel:
my $p = Net::OpenSSH::Parallel->new(workers => 5); for my $host (@hosts) { $p->add_host($host); } $p->all(rsync_put => $src, $dst); $p->run;

Replies are listed 'Best First'.
Re^2: Fork multi processes
by saurya1979 (Initiate) on May 04, 2012 at 08:26 UTC
    Thank you Salva. Is it possible to not use any external Perl module and use build-in modules to achieve it?
      The entire point of Perl is that you do not have to. There are no points to be earned for laboriously doing what has already been done (better) by someone else, such that all you need to do to solve your problem is to install something and then write five or ten additional lines in order to do it.
      Yes, sure, you can achieve it, though it would take you a while to do it.
        Yes I know :-) and I am looking for the way to achieve it. So far I coded like below:
        my @childs; my @allfolders => List of all folders for ( my $i=0; $i<@allFolders; $i+=5 ) { my @sourceFolders = @allFolders[$i..$i+4]; foreach my $folders ( @sourceFolders ) { my $pid = fork(); if ( $pid ) { # parent push(@childs, $pid); } elsif ($pid == 0) { EXECUTe RSYNC HERE.... } else { print "Error: Cannot fork process : $! \n",1); } } foreach ( @childs ) { waitpid($_, 0); } }
        The problem here is that it executes rsync for 5 folders at a time and wait until all of them finishes. BUT, I want to start rsync for 6th folder immediately after 1 of 5 rsync is completed.
      In my case, there is just 1 source and 1 destination server for rsync. I want to rsync directories located at different place on source server to the destination server. So I wanted to run multiple rsyncs to make it faster.
        #!/usr/bin/perl use strict; use warnings; use Net::OpenSSH::Parallel 0.12; use Net::OpenSSH::Parallel::Constants qw(OSSH_ON_ERROR_IGNORE); @ARGV == 3 or die <<EOU; Usage: $0 host src_dir dst_dir EOU my ($host, $src, $dst) = @ARGV; opendir my $dh, $src or die "unable to open $src"; my @ls = grep !/^\.{1,2}$/, readdir $dh; close $dh; my %now; sub on_error { my ($pssh, $label, $error) = @_; warn "transfer of $now{$label} by $label failed\n"; OSSH_ON_ERROR_IGNORE; } sub fetch_next { my ($pssh, $label) = @_; if (@ls) { my $file = shift @ls; $now{$label} = $file; warn "$label is copying $file\n"; $pssh->push($label, rsync_put => "$src/$file", "$dst/$file"); $pssh->push($label, sub => \&fetch_next); } } my $p = Net::OpenSSH::Parallel->new(on_error => \&on_error); $p->add_host("worker$_", $host) for 1..5; $p->all(sub => \&fetch_next); $p->run;
        Requires Net::OpenSSH::Parallel 0.12 I have just uploaded to CPAN.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://968878]
[Corion]: Oooh. I had hoped that X::Tiny would be an X11 client or X11 server, but it is an exception framework :-)
[MidLifeXis]: How many languages would that 'Ex' alias handle. ETOOCUTE

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (11)
As of 2017-02-27 11:48 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (383 votes). Check out past polls.