Re^2: Finding Nearly Identical Sets (Updated:4200/sec)

[BrowserUk]

FWIW: my fears that the transforms were not being detected were unfounded. There are 10 times as many possibilities for 1-insertion matches than there are for the deletion & substitution varieties, so it starts detecting insertions almost straight away, but the other two not so. However, once the number of known instances gets high enough, they start to show in roughly the expected proportions:

C:\test>\Perl22\bin\perl.exe 1172842.pl -N=1e6 -SAVE -LOAD > null
   1000000      From 1000000 received there were: 767637 new; 16713734
+ known; 0 deletions; 2899 insertions; 19788 transformations.
Numbers processed at a rate of 4366/second

C:\test>\Perl22\bin\perl.exe 1172842.pl -N=5e7 -SAVE -LOAD > null
  50000000      From 50000000 received there were: 36232459 new; 21068
+158 known; 0 deletions; 353102 insertions; 1173728 transformations.
Numbers processed at a rate of 4592/second

C:\test>\Perl22\bin\perl.exe 1172842.pl -N=5e7 -SAVE -LOAD > null
  50000000      From 50000000 received there were: 33583555 new; 52246
+313 known; 0 deletions; 650237 insertions; 1281578 transformations.
Numbers processed at a rate of 4767/second

C:\test>\Perl22\bin\perl.exe 1172842.pl -N=5e7 -SAVE -LOAD > null
  50000000      From 50000000 received there were: 30372628 new; 86174
+891 known; 0 deletions; 899052 insertions; 1715170 transformations.
Numbers processed at a rate of 5344/second

C:\test>\Perl22\bin\perl.exe 1172842.pl -N=5e7 -SAVE -LOAD > null
  50000000      From 50000000 received there were: 27148533 new; 11871
+9247 known; 93783 deletions; 1103187 insertions; 2202499 transformati
+ons.
Numbers processed at a rate of 5900/second

Note also that this code really loves having the system to itself -- the last couple of runs were overnight when the machine was otherwise idle -- and the performance jumps up from 4200/s to 5900/s; +40%.