Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Using Net::FTP::Throttle

by joetesta (Novice)
on Aug 01, 2012 at 21:31 UTC ( #984897=perlquestion: print w/ replies, xml ) Need Help??
joetesta has asked for the wisdom of the Perl Monks concerning the following question:

Hello PerlMonks,

I have an automated script using Net::FTP to upload large files and I saw from MRTG graphs that it's transferring at a rate of about 80Mbps. Since our commitment is only 50Mbps, and we don't want to get hit with overage charges, I tried to find a way to limit the upload speed to somewhere around 50Mbps.

I tried implementing Net::FTP::Throttle like this:

my $ftp = Net::FTP::Throttle->new($host, Debug => 0, Passive => 1, Bytes_read => 4096, BlockSize => 4096, MegabitsPerSecond => 50) or die "Cannot connect: $@";
But it is maxing out at about 6.5Mbps - no matter what values I try for Bytes_read, BlockSize, and MegabitsPerSecond, MRTG shows a max upload rate of about 6.5Mbps. This is too slow, does anyone know why this might be happening or another good way to limit FTP script to 50mbps?

Thank you very much in advance. With kind regards, joe

Comment on Using Net::FTP::Throttle
Download Code
Re: Using Net::FTP::Throttle
by joetesta (Novice) on Aug 01, 2012 at 22:59 UTC
    Apologies if it's bad form to answer my own post - I found that I'm getting a faster rate by decreasing rather than increasing the Bytes_read and BlockSize. With those set to 1024, I'm seeing 10Mbps+ This is good, please consider my issue resolved.
Re: Using Net::FTP::Throttle
by BrowserUk (Pope) on Aug 01, 2012 at 23:21 UTC

    I suspect that the setting of the maximum burst rate to 10k (10240) here (in new()):

    my $mbps = $arg{MegabitsPerSecond} || croak "No MegabitsPerSecond +passed"; my $bps = $mbps * 1024 * 1024 / 8; my $bucket = new Algorithm::TokenBucket $bps, 10240;

    Combined with the minimum accounting period of 1/100th of a second here (in get()):

    sleep 0.01 until $bucket->conform($len); $bucket->count($len);

    Means that the maximum throughput possible is 10kbytes per 0.001 of a second = 1000KBytes or (roughly) 10Mbps.

    Throw in the out-of-band overheads (writing to the local file), and that probably explains why it has a fixed upper limit regardless of your requested limit.

    Two (3) possible solutions:

    1. Increase the max-burst value.

      Try doubling it and see what through put it gives you. Adjust accordingly.

    2. Decrease the accounting period.

      Try halving it. Adjust ...

    3. A mix of the two.

    Also, 4096 byte blocksize seems very small for transferring large files across a network? I try 64k and see what affect that has.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Thanks BrowserUK!

      With the max-burst doubled, I was seeing better throughput. So I doubled it again and it looks like now I'm seeing around 50mbps

      Regarding the block size, I thought the same, however it seemed that anything over 4K was not helping, and I only got better throughput by decreasing it to 1024 and finally 512. Perhaps now that I've increased max-burst, a larger block size wouldn't hurt?

      at least this is working now; I wonder if Net::FTP::Throttle should be updated to increase that max-burst? How does stuff like that get done, should I be prompting someone or going at it myself?

      kind regards, Joe
        How does stuff like that get done, should I be prompting someone or going at it myself?

        I'd start by sending the author an email pointing to this thread at perlmonks and go from there.

        With regard to blocksize. I just did some not-very-representative testing using the loopback address on my machine and with 4k blocks the maximum throughput I can get is 140MB/s:

        C:\test>sock-rated.pl -O=4 -I=4 1000 svr: request for 1048576000 1048576000 took 6.999 seconds. [142.89 MB/s]

        By using 100k blocks, I can get that up to over 300MB/s:

        C:\test>sock-rated.pl -O=100 -I=100 10000 svr: request for 10485760000 10485760000 took 33.032 seconds. [302.74 MB/s]

        Not a real-world test, but still 4k seems like a pifflingly small amount.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Re: Using Net::FTP::Throttle
by flexvault (Parson) on Aug 03, 2012 at 11:52 UTC

    joetesta,

    On a lan without going through a router, large block sizes will help, but once you use the Internet, the large blocks will cause fragmentation. Try using 1492 which is the most common packet size for routers on the Internet. If you are on a GigE Internet connection, I believe 14,920 can be used, but realize the first router that can't support that size will cause fragmentation.

    Also realize that once more than one down-load is in progress, you may go over your bandwidth limit. On all of my Internet facing servers, I use the open source 'pure-ftpd' which allows bandwidth to be throttled by user, group, time of day, etc. You can chain-root the ftp session.

    If you can't chance the server, then try to limit the number of concurrent ftp sessions and limit each individual ftp session to a value less than the max usage divided by the number of concurrent sessions.

    Good Luck!

    "Well done is better than well said." - Benjamin Franklin

      Hello All, I am using the Net::Ftp for file transfer and was wondering if there was any way to display the throughput of the transfer using some FTP commands? Thanks in advance ! Arun

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://984897]
Approved by bulk88
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2014-09-22 07:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (182 votes), past polls