Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^8: Fast common substring matching

by GrandFather (Cardinal)
on Aug 25, 2005 at 21:46 UTC ( #486734=note: print w/ replies, xml ) Need Help??


in reply to Re^7: Fast common substring matching
in thread Fast common substring matching

Extrapolating from BrowserUk's estimate in Re^5: Search for identical substrings (58 hours for the 300/3k data set) and my estimate that this code is about 7000 times faster than that, the total run time should be of the order of 30 seconds. If it is more than an hour something is very wrong. Even if it is more than a few minutes our understanding is flawed or there is a bug that wasn't shown by the six string data set.


Perl is Huffman encoded by design.


Comment on Re^8: Fast common substring matching
Re^9: Fast common substring matching
by bioMan (Beadle) on Aug 26, 2005 at 15:52 UTC

    I have done a quick test. Quick because of your blazing fast algorithm. My runtimes for 3 to 200 strings is given below. The times come from the Time::HiRes::gettimeofday function in your program. I set $subStrSize = 256.

    3,0.001355 4,0.002366 5,0.003744 6,0.005187 7,0.006808 8,0.008807 9,0.011459 10,0.014172 11,0.017328 12,0.020473 13,0.023491 14,0.027905 15,0.031813 16,0.036426 17,0.043177 18,0.067228 19,0.052098 20,0.059449 30,0.132611 40,0.235241 50,0.367828 60,0.510851 70,0.695534 80,0.913879 90,1.178992 100,1.427899 125,2.254988 150,3.269179 200,5.831225

    These results are from a second copy of your program that I created. It appears my first copy still has a bug.

    Yes, there must be a bug in my original copy. Running my full data set with my backup copy of your program gave a runtime of 14.759568. I'd say that's a little better than three years! I'll check the results to see how they compare with some of my old results from limited data sets. That check could take a little time :-)

    The problem could be with the way PerlIDE runs scripts. All the times above are from the command line (Win2000).

    I ran the full data set with the second copy of the program from PerlIDE and got a runtime of 15.147881. So the problem is with my first copy of your program.

    I really like your algorithm. It's speed will allow me to modify my original data and test a couple of theories. You've got a real winner.

    update

    I checked the results from the program and the algorithm isn't finding all the matches. I made a modification which seems to do better.

    I added a new variable called $filterSize. This is used to exclude matches below a minimum length for output. It replaces $subStrSize. $subStrSize is now used to set the minimum string search size. The statement used to output the results is also modified.

    my $subStrSize = 128; my $filterSize = 256; . . . $localBest[4] > $filterSize ? printf OUT "%10s::%-10s Length[%4d] Starts(%4d %4d)\n", $strings[$localBest[0]][0], $strings[$localBest[1]][0], $loc +alBest[4], $localBest[2], $localBest[3] : print OUT " Didn't find LCS for $sourceName and $targetName\n +";

    The last statement replaces:

    @localBest ? printf OUT "%10s::%-10s Length[%4d] Starts(%4d %4d)\n", $strings[$localBest[0]][0], $strings[$localBest[1]][0], $loc +alBest[4], $localBest[2], $localBest[3] : print OUT " Didn't find LCS for $sourceName and $targetName\n +";

    I hoped this would give me better results. I used $filterSize = 256. I set $subStrSize to 16, 32, 64, or 128. The runtimes for $subStrSize were 250.505445, 124.928021, 57.274645, and 32.787612, respectively. I got the best results with $subStrSize = 128, $filterSize = 256. I don't know why $subStrSize = 128 gave better results than shorter lengths. Also $subStrSize = 256 gave significantly poorer results than any of the shorter lengths.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://486734]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2014-08-30 14:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (293 votes), past polls