Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Challenge: Fast Common Substrings

by BrowserUk (Pope)
on Apr 05, 2007 at 15:23 UTC ( #608499=note: print w/ replies, xml ) Need Help??


in reply to Challenge: Fast Common Substrings

Okay here is a benchmark as requested. I doubt it will satisfy everyone.

Some comments on the benchmark and results obtained.

  1. Randomly generated strings of 30 .. 50 lower case characters rarely if ever produce common substrings of 3 characters, never mind longer.

    To this end I used 'A'..'D', which produces a few at 3, 4, & 5 characters long, but never longer that I saw.

  2. Even adjacent pairings of 5000 strings takes a very long time to run, never mind a full combinatorial pairing.

    I used 500 strings instead. It's a command line parameter, if you've the time and inclination to run 5000, go ahead. I doubt it will make any huge difference to the outcome.

    theZip's (TZ) code is horrible to benchmark as it relies upon a global var. Ie. The sub does not return the result.

    I had a half hearted attempt to address this using a helper sub closing over the global, but the result are wildly inaccurate. Possibly? the fault of my adaption. Sorry thezip.

    However, it seems to be far and away the slowest algorithm anyway.

  3. eric256's (eric) code produces numbers that seem not to relate to the inputs as far a I can discern.

    Maybe I screwed up, but I don't think so. It's also the second slowest algorithm.

  4. Mine (Buk), blokhead's (BH1/BH2), moron's (MN), and Limbic~Region's (LR) code all produce the same counts (although Limbic~Region's has been seen to be one shy of the others on a few occasions).

    The level of concurrence between these disparate algorithms and implementations is taken to indicate that they are producing the correct results. I have not manually verified them.

Results (500 random strings of 'A'..'D', tested each against the next for common substrings 3-7 characters

c:\test>608174-b -N=500 -LENGTH=3 (warning: too few iterations for a reliable count) Rate TZ Eric BH1 MN BH2 L~R Buk TZ 9.06e-002/s -- -98% -98% -99% -99% -99% -100% Eric 4.15/s 4481% -- -19% -41% -63% -75% -85% BH1 5.12/s 5548% 23% -- -27% -55% -70% -82% MN 7.02/s 7642% 69% 37% -- -38% -58% -75% BH2 11.3/s 12354% 172% 121% 61% -- -33% -59% L~R 16.9/s 18532% 307% 230% 141% 50% -- -39% Buk 27.7/s 30457% 567% 441% 295% 145% 64% -- ---- BH2 => 14 L~R => 14 Eric => 31 Buk => 14 BH1 => 14 TZ => 393262 MN => 14 c:\test>608174-b -N=500 -LENGTH=4 (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) Rate TZ Eric BH1 MN L~R BH2 Buk TZ 9.01e-002/s -- -97% -98% -98% -99% -99% -100% Eric 2.87/s 3079% -- -40% -52% -83% -84% -88% BH1 4.78/s 5198% 67% -- -20% -71% -73% -80% MN 6.00/s 6556% 109% 26% -- -64% -65% -75% L~R 16.5/s 18153% 474% 245% 174% -- -5% -33% BH2 17.4/s 19167% 506% 264% 189% 6% -- -29% Buk 24.5/s 27060% 754% 413% 308% 49% 41% -- ---- BH2 => 5 L~R => 5 Eric => 34 Buk => 5 BH1 => 5 TZ => 396184 MN => 5 c:\test>608174-b -N=500 -LENGTH=5 (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) Rate TZ Eric BH1 MN L~R BH2 Buk TZ 9.10e-002/s -- -96% -98% -98% -99% -100% -100% Eric 2.23/s 2352% -- -54% -63% -86% -90% -91% BH1 4.85/s 5227% 117% -- -20% -70% -77% -80% MN 6.06/s 6551% 171% 25% -- -63% -72% -75% L~R 16.4/s 17953% 636% 239% 171% -- -23% -33% BH2 21.3/s 23338% 856% 340% 252% 30% -- -13% Buk 24.5/s 26791% 997% 405% 304% 49% 15% -- ---- BH2 => 1 L~R => 1 Eric => 35 Buk => 1 BH1 => 1 TZ => 387220 MN => 1 c:\test>608174-b -N=500 -LENGTH=6 (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) Rate TZ Eric BH1 MN L~R BH2 Buk TZ 9.04e-002/s -- -95% -98% -99% -99% -100% -100% Eric 1.73/s 1814% -- -65% -72% -90% -92% -93% BH1 4.92/s 5344% 184% -- -20% -70% -79% -81% MN 6.13/s 6687% 255% 25% -- -63% -73% -76% L~R 16.7/s 18321% 862% 238% 171% -- -27% -34% BH2 22.9/s 25257% 1225% 366% 274% 38% -- -9% Buk 25.2/s 27829% 1359% 413% 312% 52% 10% -- ---- BH2 => 0 L~R => 0 Eric => 35 Buk => 0 BH1 => 0 TZ => 391688 MN => 0 c:\test>608174-b -N=500 -LENGTH=7 (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) Rate TZ Eric BH1 MN L~R BH2 Buk TZ 9.08e-002/s -- -94% -98% -99% -99% -100% -100% Eric 1.47/s 1521% -- -70% -76% -91% -94% -94% BH1 4.93/s 5327% 235% -- -21% -71% -80% -81% MN 6.22/s 6748% 322% 26% -- -63% -74% -76% L~R 16.9/s 18554% 1051% 244% 172% -- -30% -33% BH2 24.1/s 26469% 1539% 390% 288% 42% -- -5% Buk 25.4/s 27899% 1627% 416% 309% 50% 5% -- ---- BH2 => 0 L~R => 0 Eric => 32 Buk => 0 BH1 => 0 TZ => 393468 MN => 0

The benchmark code. CLI parameters are: -N=nn numbers of strings to generate; -LENGTH=mm: length of common substrings to look for; Interesting challenge Limbic~Region, thanks.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re: Challenge: Fast Common Substrings
Select or Download Code
Re^2: Challenge: Fast Common Substrings
by BrowserUk (Pope) on Apr 05, 2007 at 15:34 UTC

      /me benchmarks agian with his in there as a matter of pride.

      C:\Perl\test>substr_bench.pl -N=1000 -LENGTH=2 Rate BH2 BH1 MN Eric L~R Buk BH2 3.37/s -- -20% -53% -70% -78% -87% BH1 4.21/s 25% -- -42% -62% -72% -84% MN 7.22/s 114% 71% -- -35% -52% -73% Eric 11.1/s 231% 164% 54% -- -26% -58% L~R 15.1/s 347% 258% 109% 35% -- -44% Buk 26.7/s 694% 534% 270% 140% 77% -- ---- BH2 => 14 L~R => 12 Eric => 14 Buk => 14 BH1 => 14 MN => 14 C:\Perl\test>substr_bench.pl -N=1000 -LENGTH=3 Rate BH1 MN BH2 Eric L~R Buk BH1 3.88/s -- -23% -39% -46% -73% -80% MN 5.05/s 30% -- -21% -30% -65% -74% BH2 6.40/s 65% 27% -- -11% -56% -67% Eric 7.21/s 86% 43% 13% -- -50% -63% L~R 14.4/s 272% 185% 125% 100% -- -26% Buk 19.4/s 400% 284% 203% 169% 35% -- ---- BH2 => 9 L~R => 9 Eric => 9 Buk => 9 BH1 => 9 MN => 9 C:\Perl\test>substr_bench.pl -N=1000 -LENGTH=4 Rate BH1 MN Eric BH2 L~R Buk BH1 3.56/s -- -19% -43% -57% -74% -79% MN 4.39/s 23% -- -30% -47% -68% -74% Eric 6.22/s 75% 42% -- -25% -54% -63% BH2 8.35/s 135% 90% 34% -- -38% -51% L~R 13.5/s 281% 209% 118% 62% -- -20% Buk 16.9/s 375% 285% 171% 102% 25% -- ---- BH2 => 4 L~R => 4 Eric => 4 Buk => 4 BH1 => 4 MN => 4 C:\Perl\test>substr_bench.pl -N=1000 -LENGTH=5 Rate BH1 MN Eric BH2 L~R Buk BH1 3.82/s -- -14% -34% -59% -72% -78% MN 4.44/s 16% -- -24% -52% -67% -74% Eric 5.82/s 52% 31% -- -37% -57% -66% BH2 9.27/s 143% 109% 59% -- -32% -46% L~R 13.6/s 255% 205% 133% 46% -- -21% Buk 17.1/s 348% 285% 194% 85% 26% -- ---- BH2 => 0 L~R => 0 Eric => 0 Buk => 0 BH1 => 0 MN => 0 C:\Perl\test>substr_bench.pl -N=1000 -LENGTH=6 (warning: too few iterations for a reliable count) Rate BH1 MN Eric BH2 L~R Buk BH1 3.56/s -- -21% -38% -64% -73% -80% MN 4.50/s 27% -- -21% -54% -66% -74% Eric 5.73/s 61% 27% -- -42% -57% -67% BH2 9.84/s 177% 119% 72% -- -25% -43% L~R 13.2/s 271% 193% 130% 34% -- -24% Buk 17.4/s 389% 286% 203% 77% 32% -- ---- BH2 => 1 L~R => 1 Eric => 1 Buk => 1 BH1 => 1 MN => 1

      ___________
      Eric Hodges
Re^2: Challenge: Fast Common Substrings
by eric256 (Parson) on Apr 06, 2007 at 20:07 UTC

    Hey

    I misread the definition of substring to mean all substrings of that length or lower...quick easy fix and it starts placing decent.

    sub common_sub { my ($s1,$s2,$len) = @_; my $len_s1 = length($s1); my $len_s2 = length($s2); my $match_s1 = {}; my $match_s2 = {}; for my $start (0..length($s1)-1) { #for my $l (1..$len) { next if $start+$len > $len_s1; $match_s1->{substr($s1, $start, $len)} ||= 1; #} } for my $start (0..length($s2)-1) { #for my $l (1..$len) { next if $start+$len > $len_s2; $match_s2->{substr($s2, $start, $len)} ||= 1; #} } $match_s1->{$_}++ for keys %$match_s2; return grep { $match_s1->{$_} == 2 } keys %$match_s1; }

    ___________
    Eric Hodges

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://608499]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2014-09-20 18:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (160 votes), past polls