Detecting whether two pathes refer to the same file

rovf has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Detecting whether two pathes refer to the same file by ambrus (Abbot) on Sep 10, 2010 at 09:37 UTC
The inode number alone is not enough, you also have to compare the device number from stat, because the inode numbers are unique only within each filesystem. The dev-inode pair usually identifies the file uniquely, but of course there are some caveats. Perls are normally compiled with 64-bit file access, and you might need that for this trick too, because it makes inode numbers wider. Device numbers might not be constant over a reboot. (They often are, but not always.) Under linux, if you see a file through a bind mount, it gets the same device-inode pair as the file on the original mount, so it's possible to have two file names that refer to the same underlying file but one of them is read-only. Probably more stuff I don't know about.	[reply]
Re^2: Detecting whether two pathes refer to the same file by rovf (Priest) on Sep 10, 2010 at 09:50 UTC
Thank you for the quick reply! Perls are normally compiled with 64-bit file access, and you might need that for this trick too, because it makes inode numbers wider. In what way could this affect me? I just have to compare the inode numbers (and device numbers, as you pointed out), as they are returned from stat,isn't it? Device numbers might not be constant over a reboot. As I apply the call to stat to both variables in succession, there is no possibility to be interrupted by a reboot. -- Ronald Fischer <ynnor@mm.st>	[reply] [d/l]
Re^3: Detecting whether two pathes refer to the same file by wazoox (Prior) on Sep 10, 2010 at 10:46 UTC
In what way could this affect me? I just have to compare the inode numbers (and device numbers, as you pointed out), as they are returned from stat,isn't it? If your perl doesn't support 64 bits inodes, stat will fail in weird ways; in the worst case, it could return erroneous inodes and wrongly state different files as the same.	[reply]
Re: Detecting whether two pathes refer to the same file by BioLion (Curate) on Sep 10, 2010 at 09:40 UTC
A quick look on CPAN turns up File::Same and File::is, both of which do file comparisons (Including checking if it is the 'same' file in different locations), so specific mention of symlinks, but they use the INODE approach for comparisons so it should be workable. File::is is much newer though. I haven't tested/used either, and there may well be others, but this seems to be a problem a lot of other people have thought about, so I am sure a good solution exists out there! Hope this helps! Just a something something...	[reply]
Re^2: Detecting whether two pathes refer to the same file by rovf (Priest) on Sep 10, 2010 at 10:42 UTC
A quick look on CPAN turns up File::Same and File::is File::Same doesn't check, whether two pathes name the same file, but whether two pathes point to (possibly different) files, having the same content. File::is tries to verify the identity of the files, but only by inode number, which is (as we have learned from ambrus is not even correct on Unix, and does not work at all on Windows :-( -- Ronald Fischer <ynnor@mm.st>	[reply] [d/l]
Re^2: Detecting whether two pathes refer to the same file by rovf (Priest) on Sep 10, 2010 at 09:53 UTC
Thanks a lot, I will have a look at them! -- Ronald Fischer <ynnor@mm.st>	[reply] [d/l]
Re: Detecting whether two pathes refer to the same file by BrowserUk (Patriarch) on Sep 10, 2010 at 09:57 UTC
Inode numbers don't have any meaning and are always 0 on Windows. There is an equivalent unique identifier available, (the FileIndexHigh/Low fields in the structure returned by GetFileInformationByHandle(), but you'd need to use Win32::API or Inline::C to get at it. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply]
Re: Detecting whether two pathes refer to the same file by cdarke (Prior) on Sep 10, 2010 at 10:53 UTC
On Windows NTFS there is a similar concept to an inode number, but it is a 64-bit index number split into two 32-bit numbers "Low" and "High". Like inodes, these numbers are only unique within a partition, so you also need to take into account the Volume Serial Number (VSN). See Win32::IdentifyFile. The information returned by IdentifyFile is Volume Serial Number, File Index High, File Index Low. I am the author, so let me know if you have any issues. On FAT I don't think you have any options like this.	[reply]
Re^2: Detecting whether two pathes refer to the same file by afoken (Chancellor) on Sep 10, 2010 at 11:10 UTC
On FAT I don't think you have any options like this. Right. While it is possible to create a hardlink within a FAT filesystem using a disk editor, most -- if not all -- FAT filesystem checkers consider this a filesystem error. FAT is defined in a way that each file must have exactly one directory entry, and so the fully qualified file name (relative to the filesystem) uniquely identifies a file. With VFAT (i.e. FAT + long names), things become a little bit different, because each path element may have a "long" and a "short" name. Normalizing the paths of two files to use only "short" or "long" elements gives compareable, unique identifiers. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]
Re^2: Detecting whether two pathes refer to the same file by rovf (Priest) on Sep 10, 2010 at 11:03 UTC
Would your module also work on files which reside on, say, Linux, but are accessed from Windows via WinNFS, Samba or CIFS? This wouldn't be NTFS either, would it? -- Ronald Fischer <ynnor@mm.st>	[reply] [d/l]
Re^3: Detecting whether two pathes refer to the same file by cdarke (Prior) on Sep 10, 2010 at 11:21 UTC
I have never tried it, and don't currently have a working Samba link to help. I doubt it, but I have been surprised before. I wouldn't like to bet one way or the other. Let me know if you try it!	[reply]
Re: Detecting whether two pathes refer to the same file by kejohm (Hermit) on Sep 10, 2010 at 13:20 UTC
If you are looking to test whether the two paths are referring to the same location (as opposed to the two paths referring to two separate, but identical, files) you could use Cwd::abs_path() on both, then test whether the paths returned are the same. Example: `#!perl use strict; use warnings; use feature qw(:5.12); use Cwd qw(abs_path); say abs_path ('/foo') eq abs_path ('/bar/../foo');` [download] You could also check out some of the File::* modules for working with files, directories, paths, etc. Update: Link fixed.	[reply] [d/l]
Re^2: Detecting whether two pathes refer to the same file by ikegami (Patriarch) on Sep 10, 2010 at 16:14 UTC
It's a good solution, but I can think of two limitations. That will work for symbolic links (on most devices), but not for hard links (other than "." and ".."). `$ echo foo > a $ ln a b $ cat b foo $ perl -MCwd=abs_path -E'say abs_path($_) for @ARGV' a b /tmp/a /tmp/b` [download] You might be able to check `stat`'s device plus inode fields to address this limitation on some devices on some systems. It also won't necessarily work across devices (since you could access the same file via two devices). The following all refer to the same file: `C:\Temp\file \\?\C:\Temp\file # Via UNC path \\localhost\C$\Temp\file # Via localhost \\tribble\C$\Temp\file # Via domain name \\10.0.0.6\C$\Temp\file # Via IP address \\localhost\share\file # Via share Z:\file # Given subst Z: C:\Temp` [download] In general, this isn't solvable.	[reply] [d/l] [select]
Re^3: Detecting whether two pathes refer to the same file by kejohm (Hermit) on Sep 11, 2010 at 12:43 UTC
Good points. It depends on whether or not the OP would ever come across those situations.	[reply]
Re: Detecting whether two pathes refer to the same file by afoken (Chancellor) on Sep 10, 2010 at 11:53 UTC
It's not necessary that it works for symlinks too (though, of course, it would be nice). Wrong. It must work for symlinks or it will easily break on non-windows systems. Normalising paths with symlinks is relatively easy. In Java, there is a function JavaFileManager.isSameFile, which performs exactly this function I don't think so. JavaFileManager is an interface, not a class. No code, just conventions. And by the way, JavaFileManager.isSameFile() is a method, not a function. It would require a lot of knowledge about the underlying operating system and the used filesystems. And it would get really complicated with files on network file systems. From the documentation at http://download.oracle.com/javase/6/docs/api/javax/tools/JavaFileManager.html#isSameFile%28javax.tools.FileObject,%20javax.tools.FileObject%29: `isSameFile boolean isSameFile(FileObject a, FileObject b) Compares two file objects and return true if they represent the sa +me underlying object. Parameters: a - a file object b - a file object Returns: true if the given file objects represent the same underlying o +bject Throws: IllegalArgumentException - if either of the arguments were cre +ated with another file manager and this file manager does not support + foreign file objects` [download] The method compares objects, not files. It may throw an exception when the objects weren't created by the same file manager, giving you no useful information at all. The interface makes no statement about network file systems. What happens when a filesystem is mounted in two different places, e.g. /mnt/a and /mnt/b? Will the method detect that /mnt/a/foo and /mnt/b/foo are the same file? Even if /mnt/a is a directory exported via NFS from one server and /mnt/b is the same directory exported via CIFS from the same server? Even if the NFS server and the CIFS server use different names, different network interfaces and different addresses? Another problem is that each part of the filename may be case sensitive, case preserving, or case insensitive, depending on the filesystems. File::Spec does not take that problem into account and assumes, for example, that all filenames on Unix derivates are always case sensitive (see Re^5: Unify windows filenames). So whenever you cross a filesystem border, the rules for normalising and comparing may change, depending on the filesystem itself and the mount options. Regarding the Windows API, Re: Detecting whether two pathes refer to the same file shows you a way to handle local NTFS systems. I don't know how Windows handles files on other file systems. Does it emulate the NTFS IDs like Linux does when it mounts a FAT file systems? What about ISO9660 (CDROM) and UDF (DVD)? What about network filesystems? Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply] [d/l]
Re: Detecting whether two pathes refer to the same file by ambrus (Abbot) on Sep 10, 2010 at 21:12 UTC
<joke> $ cat samefile.c /* samefile -- decides whether two files given as arguments are the sa +me. follows symlinks. both files must be valid and you must have permissio +n to read them, and even then it may fail with an error or hang. exit code 0 is same, 1 if different, higher on an error. / #include <unistd.h> #include <fcntl.h> #include <sys/types.h> #include <sys/wait.h> int main(int argc, char argv[]) { int d0, d1, sch; struct flock lk; pid_t ppa, pch; if (3 != argc) return 10; if (-1 == (d0 = open(argv[1], O_WRONLY))) return 4; if (-1 == (d1 = open(argv[2], O_RDONLY))) return 3; lk.l_type = F_WRLCK; lk.l_whence = SEEK_SET; lk.l_start = 0; lk.l_len = 1; if (-1 == fcntl(d0, F_SETLK, &lk)) return 5; ppa = getpid(); if (-1 == (pch = fork())) return 9; if (!pch) { lk.l_type = F_RDLCK; lk.l_whence = SEEK_SET; lk.l_start = 0; lk.l_len = 1; if (-1 == fcntl(d1, F_GETLK, &lk)) return 4; return !(F_WRLCK == lk.l_type && ppa == lk.l_pid); } else { if (pch != wait(&sch)) return 8; if (!WIFEXITED(sch)) return 7; return WEXITSTATUS(sch); } } $ gcc -Wall -O -o samefile samefile.c $ ./samefile /usr/local/libexec/git-core/git-{diff,pull}; echo $? 1 $ ./samefile /usr/local/libexec/git-core/git-{diff,commit}; echo $? 0 $ [download] </joke> `$ test /usr/local/libexec/git-core/git-diff -ef /usr/local/libexec/git +-core/git-pull; echo $? 1 $ test /usr/local/libexec/git-core/git-diff -ef /usr/local/libexec/git +-core/git-commit; echo $? 0` [download] NOTE: test -ef won't follow symlinks Update: exercise to the reader: explain why the fork is necessary in the above program.	[reply] [d/l] [select]


Keep It Simple, Stupid
	PerlMonks