Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

  1. MD5 scales linearly (so does SHA-1, by the way).
  2. You will trade reliability for speed, I'm afraid. While adding up all the bytes is also a linear algorithm, it's a faster algorithm, but quite unreliable for your purposes.

Rather than skipping large files, just limit the size of the portion of the file you calculate an MD5 on. Limit it to, for example, 2^16. You will have false-positive collisions. But they will be relatively infrequent. For each of collisions, go back and hash the entire file.

Now in the unlikely event that you get a second round of collisions, you can do a byte by byte comparison. So the steps would be:

  1. For each file read the first 64KB or 128KB ($/ will be helpful) and generate a hash.
  2. For each collision read the entire file and generate a hash.
  3. For each whole-file-hash collision, do a byte by byte comparison.

Git uses the SHA-1 algorithm to uniquely identify commits, as well as objects under its purview. I think that SHA-1 was chosen over MD5 because it has a smoother distribution (less likely for pathological datasets to have a higher than average possibility of causing a collision). Though still linear, it's a slower algorithm than MD5. But if you intend to follow the suggestion of only hashing the first 64, 128, or 256k of each file on first pass, speed shouldn't be such an issue, and reliability improves to the point that you may never have to do a byte-by-byte comparison as a third pass.

If you switch to the slower SHA1 algorithm, and follow the steps above, I doubt you would ever exercise step three except for when you come across truly identical files, which would produce a collision at step three also. Theoretically it's possible to get a SHA1 false positive collision, but the distribution is pretty smooth, and 1/1000000000000000000000000000000000000000000000 is a really small probability.


In reply to Re: [OT]:Faster signature algorithm than md5? by davido
in thread [OT]:Faster signature algorithm than md5? by swampyankee

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    [Corion]: marto: Clever! ;)
    [Corion]: You can only reach me by pager
    [Corion]: Maybe the solution would be to launch a cron job every minute that takes two measurements a minute apart and sends a mail if the usage is below on the first and above threshold on the last measurement
    [marto]: that's essentially it :)
    [marto]: I think the long term solution would be to have sysadmins that do their job, so I don't have to do everything :P
    [marto]: they already have an entire BMC patrol system, which they disabled, because it was sending out spurious messages. So rather than fix the issue, or even find out what it was, they turned it off. No messages, can't be any problems, right?
    [Corion]: marto: But having open tickets / incidents increases the pressure on them ;) Of course, likely your contract / SLA specifies an upper limit for the number of incidents :-D
    [Corion]: marto: Ow ...
    [marto]: Corion They don't get into trouble for the number of tickets, or even breached tickets they have
    [marto]: which is a core issue with the client

    How do I use this? | Other CB clients
    Other Users?
    Others taking refuge in the Monastery: (10)
    As of 2017-01-24 10:14 GMT
    Find Nodes?
      Voting Booth?
      Do you watch meteor showers?

      Results (203 votes). Check out past polls.