comment on

Yes, but there is a fundamental difference...

The first script will only do MD5 hashes on files if there is more than one file with the same file size, then compares the MD5s for the files of that size. Yours MD5's *everything*, then compares *all* the MD5s. If a file has a unique filesize, it *can't* have a duplicate.

Depending on the make up of the files, this can have a dramatic effect:

Files:        15272
Duplicates:   999
Bytes:        15073525
[download]

Results:

First script:
real    0m11.855s
user    0m2.590s
sys     0m1.640s

Second script:
real    0m49.589s
user    0m17.110s
sys     0m6.500s
[download]

The second script is four times slower than the first...

Admittedly, if all your files were the same size there would be no difference, but in most cases, the first script will win. But hey...

In reply to Re: Re: Find duplicate files. by Anonymous Monk
in thread Find duplicate files. by salvadors

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


The stupid question is the question not asked
	PerlMonks