Recursive search for duplicate files

[props]

Dear Monks I write a script which aims to search through a directory tree and report duplicate folders and files. At the moment I'm stuck and i hope you to point me to the right direction. I could straggle with the syntax as long as i have a sort of pseudo code:

use warnings;
use strict;
use File::Find;

my $directory = '/mnt/music/very-good';

find (\&wanted, $directory);
 
  my %path_file;
  sub wanted {
     my  $path = $File::Find::dir;
     my  $filename = $File::Find::name;
 
        $hash{$path} = $filename;
  }
  my %count;
  while (my ($key , $value) = each(%path_file)) {
 
      $count{$key} +=1;
 
  }

[moritz]

What are "duplicate" files for you? Files with the same basename? Or with with identical contents?

It seems that you are using too many hashes, that are initialized nowhere: %hash is used but never declared, %path_file used and neither declared nor initialized.

[hawtin]

Have you seen Scanning for duplicate files and Find Duplicate Files and the CPAN module:

File::Find::Duplicates

[sh1tn]

Much better way is to use MD5 for file comparison.

[moritz]

If used naively, that doesn't work out well for large files, because they have to be read from disc entirely.

If you care about performance, you might just want to hash the first 5% (or the first 1k or whatever) and see if there are any collisions, and if there are you can still look at the entire file.

[sh1tn]

I can agree. Another measure, taking in mind the performance, can be the filesize comparison before all.

[props]

Well file and folder names are sufficient.

use warnings;
use strict;
use File::Find;
  
  
   my $directory = '/mnt/music/very-good';
  
   find (\&wanted, $directory);
  
  my %path_file;
 
  sub wanted {
    my  $path = $File::Find::dir;
    my  $filename = $File::Find::name;
 
       $path_file{$path} = $filename;
 
  my %count;
  while (my ($key , $value) = each(%path_file)) {
 
      $count{$key} +=1;
 
         }
  }

[moritz]

There you go: the idea is to store for each filename in which paths it occurs.

#!/usr/bin/perl
use warnings;
use strict;
use File::Find;
use File::Spec;
  
  
my $directory = shift @ARGV || '/mnt/music/very-good';
  
find (\&wanted, $directory);

my %path_file;

sub wanted {
    my $path = $File::Find::dir;
    my $filename = File::Spec->abs2rel($File::Find::name, $path);

    push @{$path_file{$filename}}, $path;

}

while (my ($filename, $paths) = each %path_file){
    if (scalar @$paths >= 2){
        print "$filename occurs in these paths: ", join(", ", @$paths)
+, "\n";
    }
#    else {
#        print "$filename is uniq\n";
#    }
}

[props]

I have a question: Since %path_file is a hash why becomes an array here:

 push @{$path_file{$filename}}, $path;

Dereferencing the array @path_file or the hash %path_file

[strat]

If you want to compare files by content (and not by name), you could have a look at the search and compare algorithm of http://www.fabiani.net/ -> Perl -> Downloads -> FindDuplicateFiles. It only runs under Win32 (because I only need it under Win32), but it is easy to port.