|Perl: the Markov chain saw|
What kind of file is it? For example is it a log file for some program? Is it a file that gets changed often or do you download it every day just in case it was changed yesterday? Do you need all the date contained in it?
With answers to these questions we might be able to help reduce the times you need to download it at all, and that might make your benefactors very happy.
A few scenarios come to mind:
Let's consider these options:It's a file which only changes rarely
Run a hashing algorithm over the file (eg md5sum) and download the result. If the hash is the same as the previous day then don't download it.
A disadvantage with this is you may need to ask their sysadmin to assist you somewhat, by giving you sufficent access to run this command, or by agreeing to run a scheduled task every night which creates the hash for you.It's a log file
Rotate the file more often (daily), so that you only need to download the bits you care about.It's a file which changes all the time
This is the hard one. All you can do here is:
Of course, if it is one of the first two, then so long as you compress you file as well; one of those methods should help a lot.
Hope this helps.