note BrowserUk <blockquote><i> My actual set of phrases will conform to a corpus of roughly 15,000 existing items, so there are no typos, misspellings or synonyms involved. </i></blockquote> <p>Then, I would approach the problem this way. <ol><li>Store the corpus of phrases in its own table each with a unique numeric value. </li><li>Each set of phrases then becomes a bitfield with 1-bit set in the appropriate position for each phrase that set contains. </li><li>Your similarity can then be some hueristic based on that population counts (bit count) of ANDing and XORing the two bitstrings that represent each set. <p>The population count of the result of ANDing two set's bitstrings will tell you how many phrases they have in common; <p>The population count of the result of XORing two set's bitstrings will tell you how many phrases that appear in one but not the other. <p>You can then combine those two numbers mathematically to reflect whether the sharing of phrases is more important than having phrases not in common -- or vice versa -- and come up with a single number for each pairing that you can then apply a threshold value to. </li></ol> <p>You'd need a DB that supports bitstrings -- postgresql and mysql seem to -- and AND/XOR & popcount of bitstrings. I couldn't (from a quick look) see a popcount function, but (at least in the case of PgSQL), it should be a simple thing to add a PL/Perl function to do this using Perl's <code> $popcount = unpack '%b*', $bitstring; </code> <p>Food for thought perhaps. <div class="pmsig"><div class="pmsig-171588"> <hr /> <font size=1 > <div>With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'</div> <div>Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.</div> <div>"Science is about questioning the status quo. Questioning authority". </div> <div>In the absence of evidence, opinion is indistinguishable from prejudice. <p align=right> [http://thebottomline.cpaaustralia.com.au/|RIP Neil Armstrong]</p></div> </font> </div></div> 996530 996535