Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Comparing sets of phrases stored in a database?

by sundialsvc4 (Abbot)
on Oct 01, 2012 at 15:56 UTC ( #996684=note: print w/ replies, xml ) Need Help??


in reply to Comparing sets of phrases stored in a database?

As a slight parenthetical comment to BrowserUK’s excellent (and heavily up-voted) advice, I have also found it useful in some situations to create an additional mathematical set (bit-string ...) which represents “categories” of words that are present in the result.   (The word, “category,” meaning precisely whatever-it-makes-sense-to-you for it to mean.)   The idea is that a phrase could therefore quickly be eliminated from consideration if its set of categories-present is not the same.   (If the category taxonomy contained, say, less than 32 bits, it could be represented as an integer.)   The idea being to reduce the search-space, if say some kind of database I/O is required.

The notion is 150% dependent upon the needs of this application scenario.   First of all, the notion of creating a digest at all must “make sense” in terms of the application spec ... there must be a “natural taxonomy” that the app can usefully exploit “for free.”   Then, it must be shown to called-for, and beneficial.   For instance, in some high-performance critical scenarios, it might well be contra-indicated to store the bit-strings in any sort of database at all:   even a brute-force search of data, if known to entirely fit within the process’s projected resident working-set, avoids all I/O operations entirely.   But if the data were much larger, the notion of creating a lookup digest might apply.


Comment on Re: Comparing sets of phrases stored in a database?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://996684]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (6)
As of 2014-10-31 05:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (214 votes), past polls