Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Find Duplicate Files

by salvadors (Pilgrim)
on Jan 04, 2001 at 23:08 UTC ( #49819=sourcecode: print w/replies, xml ) Need Help??
Category: Utility Scripts
Author/Contact Info Tony /
Description: As my original Find Duplicate Files script was so popular I decided to take the advice of turning it into a module. Here's the initial verison of it. I'd appreciate feedback on ways to provide a nicer, more useful, interface than just returning a HoL.



package File::Find::Duplicates;

=head1 NAME

File::Find::Duplicates - Find duplicate files


  use File::Find::Duplicates;

  my %dupes = find_duplicate_files('/basedir');

  local $" = "\n  ";
  foreach my $filesize (keys %dupes) {
    print "Duplicate files of size $filesize:\n  @{$dupes{$filesize}}\


This module provides a way of finding duplicate files on your system.

When passed a base directory (or list of such directories) it returns
a hash, keyed on filesize, of lists of the identical files of that siz

=head1 TODO

Provide some much more useful interfaces to this.

=head1 AUTHOR

Tony Bowden,

=head1 SEE ALSO



use vars qw($VERSION @ISA @EXPORT %files);

require Exporter;

@ISA     = qw/Exporter/;
@EXPORT  = qw/find_duplicate_files/;
$VERSION = '0.02';

use strict;
use File::Find;
use Digest::MD5;

sub check_file {
  -f && push @{$files{(stat(_))[7]}}, $File::Find::name;

sub find_duplicate_files {
  my %dupes;
  find(\&check_file, shift || ".");
  foreach my $size (sort {$b <=> $a} keys %files) {
    next unless @{$files{$size}} > 1;
    my %md5;
    foreach my $file (@{$files{$size}}) {
      open(FILE, $file) or next;
      push @{$md5{Digest::MD5->new->addfile(*FILE)->hexdigest}},$file;
    foreach my $hash (keys %md5) {
      push @{$dupes{$size}}, @{$md5{$hash}} 
        if (@{$md5{$hash}} > 1);
  return %dupes;

"dissolving ... removing ... there is water at the bottom of the ocean
Replies are listed 'Best First'.
Re: Find Duplicate Files
by merlyn (Sage) on Jan 04, 2001 at 23:48 UTC
    I don't like the package %files there. You can get rid of it by passing a closure to File::Find...
    sub find_duplicate_files { my %dupes; my %files; find sub { -f && push @{$files{(stat(_))[7]}}, $File::Find::name; }, shift || "."; .... }
    Much cleaner. Easier to maintain. Smaller locality of reference. Blah blah blah. {grin}

    -- Randal L. Schwartz, Perl hacker

Re: Find Duplicate Files
by kingman (Scribe) on Jul 04, 2002 at 20:56 UTC
    Hi, I wrote a command-line utility using your module that makes is easy to delete duplicate files.
    #!/usr/bin/perl -w use strict; use File::Find::Duplicates; $|++; # AutoFlush the Buffer &usage if $#ARGV eq '-1'; my %dupes = find_duplicate_files(@ARGV); die "No duplicates found!\n" unless keys %dupes; print "############ Duplicate File Report & Removal Utility ########## +##\n"; my $i = 1; foreach my $fsize (keys %dupes) { print "#" x 64 . " " . $i++ . "\n"; print map {-l $_ ? "# push \@delete, '$_'; # symlinked to " . read +link($_) . "\n": "# push \@delete, '$_';\n"} @{ $dupes{$fsize} }; print "\n"; } print "unlink \@delete;\n"; sub usage { (my $script_name = $0) =~ s#.*/##; # $0 = full path to script print <<END; Generates a Report on Duplicate Files. Usage: $script_name [List of Directories] END exit } ### POD ### =head1 Name dupes - a command line utility to report on all duplicate files, even +if they have different names. This is good for mp3s and multiple drafts of do +cuments that may have been backed up in different places. =head1 Synopsis dupes [list of directories to search recursively] =head1 From an empty buffer in Vim The following commands will fill the buffer with a report of all dupli +cate files. :%!dupes [list of directories] B<or> !!dupes [list of directories] The report generated by the above commands is yet another perl script +that can be edited allowing you to flag certain files for removal. The following command will run the report and remove all flagged files +. :%!perl Nothing is deleted unless you flag the file by uncommenting the line. If you don't understand how the report works, the following commands s +hould explain it. perldoc -f push perldoc -f unlink =head1 AUTHOR Kingsley Gordon, E<lt>kingman@ncf.caE<gt> last modified: Thu Jul 4 15:11:26 EDT 2002 =cut
      It would be nice if the script deleted the duplicates but later created a hard link from the original to the deleted file. That way you don't waste any space and you have no risk of breaking anything.
Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://49819]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2017-07-25 21:14 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (381 votes). Check out past polls.