Background downloading service

[kevin_i_orourke]

I'm thinking about writing something in Perl and need some advice, especially if someone has already done something similar.

At work we have a satellite link to the internet, at 128/64 Kb/s (downlink/uplink). Hooked on to that are a whole load of computers.

Mostly this works fine, but if somebody decides to download a big file it slows internet access down for everyone.

I want to set up a download service on our Linux server that will allow users to queue downloads. The files will then be downloaded when the link is less busy (evening or nighttime) and the user will be notified by email when the file is ready to collect.

Does anyone know if something like this already exists?

If not I'm planning to write it in Perl. I can use LDAP to authenticate our users and SNMP to find out (from our router) how busy the link is.

One thing I'm missing is a module that will do HTTP and FTP downloads and let me adjust the bandwidth after the download has started. cUrl will let me set bandwidth, but won't let me adjust it later.

Any suggestions?

[madbombX]

I know for FTP downloads, you can use Net::FTP::Throttle which is a subclass of Net::FTP to throttle your FTP connections.

To implement curl in perl, I know you can use WWW::Curl although I don't believe that the PM can do any bandwidth adjustment (even at the beginning).

Eric

[kevin_i_orourke]

Thanks for that information, it's a shame there's no Net::HTTP::Throttle.

[dtr]

I know that this isn't exactly what you asked for, but iptables in linux supports bandwidth throttling (I believe that it's called "traffic shaping"). This could allow you to stop one user taking over all of the bandwidth for other users.

For http downloads, you could also try setting up a proxy server using apache (or squid - but I am less familiar with it than apache). The mod_bandwidth module for apache will allow you to control how much bandwidth people on different machines can use. This would also allow commonly accessed files to be cached on the proxy server.

I don't know if any of the perl tools will allow you to control the bandwidth that they will use - although if you're planning to write a system to schedule downloads for outside of office hours, you may find that you just don't care, as no-one else will be using the link at this time anyway!

[kevin_i_orourke]

The iptables and proxy options would be good, except our router is a PC running the MikroTik RouterOS.

It has a built-in proxy and bandwidth shaping, both of which we're using. The problem isn't so much one user taking over all the bandwidth as taking over enough of it to annoy everyone else.

Only downloading outside office hours is troublesome because we're in Nigeria and there's often no electricity outside office hours.

I'm starting to think that the system could just start downloading when the router reports that the link is quiet, stop if it gets busier and then resume later. I'd include an overall bandwidth limit by time of day as well.

I was sort of hoping somebody had already invented this wheel...

[shmem]

You might want to have a look at the command line utility wget. It supports bandwidth limiting via the --limit-rate switch.

--shmem

[kevin_i_orourke]

I've used wget a lot for other things, the problem here is that you set the bandwidth limit at the beginning and can't change it later.

In my case I could be partway through a huge (10 hours or so) download and notice that it's time to reduce bandwidth usage.

[shmem]

wget also supports resuming partial downloads - if the server supports that, too. One could check if that is the case on startup of a session, then there should be no problem interrupting and re-starting later, if it's necessary.

--shmem

[phwaap]

Clever configuration of Squid's delay pools (http://www.squid-cache.org/Doc/FAQ/FAQ-19.html#ss19.8) might work.