Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: •Re: Interlaced duplicate file finder

by abell (Chaplain)
on Jan 07, 2003 at 00:31 UTC ( #224803=note: print w/replies, xml ) Need Help??

in reply to •Re: Interlaced duplicate file finder
in thread Interlaced duplicate file finder

Most of the complication is in place to reduce file reading to a bare minimum. Say you have two 1 Gbyte files. The size is exactly the same, but the files are very different. I wouldn't want to read and digest both files to understand they are different, when it's enough to read a few bytes in the same position. My program deals rather well with these cases. It starts by reading a small chunk from all files of the same size and uses that chunk as key to partition the group of files. If any subset contains more than one file, then read another chunk starting from another (preferably far) position and iterate.

It's more or less like the naif "real life" way of comparing things. If you have two books with a blank cover, to check if they are different you first compare the size. If it's the same, you open the same page from both and check if they differ. Only if the books are the same you need to keep on reading until the end.

Moreover, by using byte by byte comparison instead of hashing, you don't even risk false positives. As small as the risk may be, it will most surely happen for your presentation due tomorrow.

Package Finder::Looper takes care of the iteration. Each call to $looper->next returns a new pair ( start, length ) within a given range, so that consecutive calls sample from different parts of the file. That's the "interlaced" part (which I should maybe have called "interleaved", but hey! this side of the world it's not the best time for choosing names in foreign languages).

Having said this, the program probably needs some tweaking to better exploit filesystem/buffering/head-positioning optimizations.


Antonio Bellezza

The stupider the astronaut, the easier it is to win the trip to Vega - A. Tucket
  • Comment on Re: •Re: Interlaced duplicate file finder

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://224803]
[Corion]: Maybe that is the push I need to finally try out Gitprep ;)
[Corion]: On the upside, I should finally improve Image::CCV to also do ImageNet classification using their pretrained parameters
[Lady_Aleena]: Hello Corion.
[Corion]: Hi Lady_Aleena!
[Lady_Aleena]: Corion, how are things?
[Corion]: Lady_Aleena: Quite good ;) I'm working four days now, instead of five, which helps my mood and my weekends tremendously
[Corion]: My Perl output hasn't recovered, but as $work is still somewhat stressful, I don't think this would be different with a five day workdweek
[Lady_Aleena]: Corion, that is a good thing (TM) 8)
[Corion]: And this evening, I'm actually dusting off some old module of mine and bringing it up to the last version of the library I'm wrapping

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (6)
As of 2017-09-24 18:09 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (274 votes). Check out past polls.