Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Fork multi processes

by salva (Monsignor)
on May 04, 2012 at 07:45 UTC ( #968878=note: print w/ replies, xml ) Need Help??


in reply to Fork multi processes

use Net::OpenSSH::Parallel:

my $p = Net::OpenSSH::Parallel->new(workers => 5); for my $host (@hosts) { $p->add_host($host); } $p->all(rsync_put => $src, $dst); $p->run;


Comment on Re: Fork multi processes
Download Code
Re^2: Fork multi processes
by saurya1979 (Initiate) on May 04, 2012 at 08:26 UTC
    Thank you Salva. Is it possible to not use any external Perl module and use build-in modules to achieve it?
      In my case, there is just 1 source and 1 destination server for rsync. I want to rsync directories located at different place on source server to the destination server. So I wanted to run multiple rsyncs to make it faster.
        #!/usr/bin/perl use strict; use warnings; use Net::OpenSSH::Parallel 0.12; use Net::OpenSSH::Parallel::Constants qw(OSSH_ON_ERROR_IGNORE); @ARGV == 3 or die <<EOU; Usage: $0 host src_dir dst_dir EOU my ($host, $src, $dst) = @ARGV; opendir my $dh, $src or die "unable to open $src"; my @ls = grep !/^\.{1,2}$/, readdir $dh; close $dh; my %now; sub on_error { my ($pssh, $label, $error) = @_; warn "transfer of $now{$label} by $label failed\n"; OSSH_ON_ERROR_IGNORE; } sub fetch_next { my ($pssh, $label) = @_; if (@ls) { my $file = shift @ls; $now{$label} = $file; warn "$label is copying $file\n"; $pssh->push($label, rsync_put => "$src/$file", "$dst/$file"); $pssh->push($label, sub => \&fetch_next); } } my $p = Net::OpenSSH::Parallel->new(on_error => \&on_error); $p->add_host("worker$_", $host) for 1..5; $p->all(sub => \&fetch_next); $p->run;
        Requires Net::OpenSSH::Parallel 0.12 I have just uploaded to CPAN.
      Yes, sure, you can achieve it, though it would take you a while to do it.
        Yes I know :-) and I am looking for the way to achieve it. So far I coded like below:
        my @childs; my @allfolders => List of all folders for ( my $i=0; $i<@allFolders; $i+=5 ) { my @sourceFolders = @allFolders[$i..$i+4]; foreach my $folders ( @sourceFolders ) { my $pid = fork(); if ( $pid ) { # parent push(@childs, $pid); } elsif ($pid == 0) { EXECUTe RSYNC HERE.... } else { print "Error: Cannot fork process : $! \n",1); } } foreach ( @childs ) { waitpid($_, 0); } }
        The problem here is that it executes rsync for 5 folders at a time and wait until all of them finishes. BUT, I want to start rsync for 6th folder immediately after 1 of 5 rsync is completed.
      The entire point of Perl is that you do not have to. There are no points to be earned for laboriously doing what has already been done (better) by someone else, such that all you need to do to solve your problem is to install something and then write five or ten additional lines in order to do it.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://968878]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (6)
As of 2014-10-02 06:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (49 votes), past polls