Beefy Boxes and Bandwidth Generously Provided by pair Networks RobOMonk
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
What kind of file is it? For example is it a log file for some program? Is it a file that gets changed often or do you download it every day just in case it was changed yesterday? Do you need all the date contained in it?

With answers to these questions we might be able to help reduce the times you need to download it at all, and that might make your benefactors very happy.

A few scenarios come to mind:

  1. It's a file which only changes once every 2 days or less often
  2. It's a log file or similar and you only care about the last X many lines, where X is much smaller than the total number of lines
  3. It's a file which changes all the time and you care about all of it

Let's consider these options:

It's a file which only changes rarely

Run a hashing algorithm over the file (eg md5sum) and download the result. If the hash is the same as the previous day then don't download it.

A disadvantage with this is you may need to ask their sysadmin to assist you somewhat, by giving you sufficent access to run this command, or by agreeing to run a scheduled task every night which creates the hash for you.

It's a log file

Rotate the file more often (daily), so that you only need to download the bits you care about.

It's a file which changes all the time

This is the hard one. All you can do here is:

  • Make sure you compress the file
  • Shape your traffic
  • Hope they'll still like you.
  • Consider whether you can download it every two days late at night, instead of daily, so as to minimise the disruption caused.

Of course, if it is one of the first two, then so long as you compress you file as well; one of those methods should help a lot.

Hope this helps.

jarich


In reply to Re: Limiting bandwidth by jarich
in thread Limiting bandwidth by sgifford

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others exploiting the Monastery: (8)
    As of 2014-04-21 08:06 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      April first is:







      Results (492 votes), past polls