Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^3: Performance of FTP Transfer

by jrsimmon (Hermit)
on Jul 22, 2009 at 18:13 UTC ( [id://782398]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Performance of FTP Transfer
in thread Performance of FTP Transfer

Your node got me wondering at what point forking more workers fails to improve performance. This is admittedly very environment-specific, but I found it interesting nonetheless. I put together the following simple script as a test:

Environment:

  • Client running on w2k3 and connecting to a HP-UX box
  • Directory to be transferred contains 340 .pdf files ranging in size from ~10k to ~250k.

use strict; use warnings; use Net::FTP; my $kids; my @kids = (); my @fileList = glob("*.pdf"); open(OUT, ">ftp_test.out") or die; for $kids (1..30){ my @tempList = @fileList; my $start_time = time; while(my @files = splice(@tempList, 0, ($#fileList + 1) / $kids)){ my $pid = fork; push(@kids, $pid); if($pid == 0){ #kid &FTP_SUB(@files); exit; } } waitpid($_, 0) foreach (@kids); my $elapsed_time = time - $start_time; print OUT "Elapsed time with $kids kids:\t$elapsed_time seconds\n"; print "Elapsed time with $kids kids:\t$elapsed_time seconds\n"; } sub FTP_SUB { my @files = @_; my $start_time = time; my $end_time = undef; my $host = '...'; my $user = 'user'; my $pwd = 'pwd'; my $ftp = Net::FTP->new($host, Debug => 0); unless($ftp->login($user,$pass)){ warn "$$:\tCannot login " . $ftp->message; } unless($ftp->cwd("ftp_test")){ warn "$$:\tCannot change directory " . $ftp->message; } foreach my $file (@files){ unless($ftp->put($file)){ warn "$$:\tPut failed " . $ftp->message; } } $ftp->quit; $ftp = undef; }
__OUTPUT__ Elapsed time with 1 kids: 177 seconds Elapsed time with 2 kids: 92 seconds Elapsed time with 3 kids: 64 seconds Elapsed time with 4 kids: 56 seconds Elapsed time with 5 kids: 40 seconds Elapsed time with 6 kids: 34 seconds Elapsed time with 7 kids: 28 seconds Elapsed time with 8 kids: 27 seconds Elapsed time with 9 kids: 24 seconds Elapsed time with 10 kids: 23 seconds Elapsed time with 11 kids: 22 seconds Elapsed time with 12 kids: 20 seconds Elapsed time with 13 kids: 19 seconds Elapsed time with 14 kids: 18 seconds Elapsed time with 15 kids: 19 seconds Elapsed time with 16 kids: 17 seconds Elapsed time with 17 kids: 15 seconds Elapsed time with 18 kids: 17 seconds Elapsed time with 19 kids: 16 seconds Elapsed time with 20 kids: 16 seconds Elapsed time with 21 kids: 15 seconds Elapsed time with 22 kids: 14 seconds Elapsed time with 23 kids: 13 seconds Elapsed time with 24 kids: 12 seconds Elapsed time with 25 kids: 15 seconds Elapsed time with 26 kids: 13 seconds Elapsed time with 27 kids: 13 seconds Elapsed time with 28 kids: 13 seconds Elapsed time with 29 kids: 13 seconds Elapsed time with 30 kids: 13 seconds

You can see that the initial forks provided quite significant performance boosts. However, the improvement began to drop quickly once I was launching more than 7 children. Eventually there was no improvement whatsoever (though no cost, either).

Things to keep in mind:

  • These are win32 forks. A *nix system may have very different results
  • This script was written specifically for this testcase -- not meant to be very robust (but I did try to make it very readable)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://782398]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others cooling their heels in the Monastery: (2)
As of 2025-02-08 23:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Which URL do you most often use to access this site?












    Results (95 votes). Check out past polls.