Your node got me wondering at what point forking more workers fails to improve performance. This is admittedly very environment-specific, but I found it interesting nonetheless. I put together the following simple script as a test:
Environment:
-
Client running on w2k3 and connecting to a HP-UX box
-
Directory to be transferred contains 340 .pdf files ranging in size from ~10k to ~250k.
use strict;
use warnings;
use Net::FTP;
my $kids;
my @kids = ();
my @fileList = glob("*.pdf");
open(OUT, ">ftp_test.out") or die;
for $kids (1..30){
my @tempList = @fileList;
my $start_time = time;
while(my @files = splice(@tempList, 0, ($#fileList + 1) / $kids)){
my $pid = fork;
push(@kids, $pid);
if($pid == 0){
#kid
&FTP_SUB(@files);
exit;
}
}
waitpid($_, 0) foreach (@kids);
my $elapsed_time = time - $start_time;
print OUT "Elapsed time with $kids kids:\t$elapsed_time seconds\n";
print "Elapsed time with $kids kids:\t$elapsed_time seconds\n";
}
sub FTP_SUB {
my @files = @_;
my $start_time = time;
my $end_time = undef;
my $host = '...';
my $user = 'user';
my $pwd = 'pwd';
my $ftp = Net::FTP->new($host, Debug => 0);
unless($ftp->login($user,$pass)){
warn "$$:\tCannot login " . $ftp->message;
}
unless($ftp->cwd("ftp_test")){
warn "$$:\tCannot change directory " . $ftp->message;
}
foreach my $file (@files){
unless($ftp->put($file)){
warn "$$:\tPut failed " . $ftp->message;
}
}
$ftp->quit;
$ftp = undef;
}
__OUTPUT__
Elapsed time with 1 kids: 177 seconds
Elapsed time with 2 kids: 92 seconds
Elapsed time with 3 kids: 64 seconds
Elapsed time with 4 kids: 56 seconds
Elapsed time with 5 kids: 40 seconds
Elapsed time with 6 kids: 34 seconds
Elapsed time with 7 kids: 28 seconds
Elapsed time with 8 kids: 27 seconds
Elapsed time with 9 kids: 24 seconds
Elapsed time with 10 kids: 23 seconds
Elapsed time with 11 kids: 22 seconds
Elapsed time with 12 kids: 20 seconds
Elapsed time with 13 kids: 19 seconds
Elapsed time with 14 kids: 18 seconds
Elapsed time with 15 kids: 19 seconds
Elapsed time with 16 kids: 17 seconds
Elapsed time with 17 kids: 15 seconds
Elapsed time with 18 kids: 17 seconds
Elapsed time with 19 kids: 16 seconds
Elapsed time with 20 kids: 16 seconds
Elapsed time with 21 kids: 15 seconds
Elapsed time with 22 kids: 14 seconds
Elapsed time with 23 kids: 13 seconds
Elapsed time with 24 kids: 12 seconds
Elapsed time with 25 kids: 15 seconds
Elapsed time with 26 kids: 13 seconds
Elapsed time with 27 kids: 13 seconds
Elapsed time with 28 kids: 13 seconds
Elapsed time with 29 kids: 13 seconds
Elapsed time with 30 kids: 13 seconds
You can see that the initial forks provided quite significant performance boosts. However, the improvement began to drop quickly once I was launching more than 7 children. Eventually there was no improvement whatsoever (though no cost, either).
Things to keep in mind:
- These are win32 forks. A *nix system may have very different results
- This script was written specifically for this testcase -- not meant to be very robust (but I did try to make it very readable)