Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
Yes, but there is a fundamental difference...

The first script will only do MD5 hashes on files if there is more than one file with the same file size, then compares the MD5s for the files of that size. Yours MD5's *everything*, then compares *all* the MD5s. If a file has a unique filesize, it *can't* have a duplicate.

Depending on the make up of the files, this can have a dramatic effect:

Files: 15272 Duplicates: 999 Bytes: 15073525
First script: real 0m11.855s user 0m2.590s sys 0m1.640s Second script: real 0m49.589s user 0m17.110s sys 0m6.500s
The second script is four times slower than the first...

Admittedly, if all your files were the same size there would be no difference, but in most cases, the first script will win. But hey...

In reply to Re: Re: Find duplicate files. by Anonymous Monk
in thread Find duplicate files. by salvadors

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others surveying the Monastery: (6)
    As of 2019-05-23 00:40 GMT
    Find Nodes?
      Voting Booth?
      Do you enjoy 3D movies?

      Results (140 votes). Check out past polls.

      • (Sep 10, 2018 at 22:53 UTC) Welcome new users!