Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re: Comparing sets of phrases stored in a database?

by sundialsvc4 (Abbot)
on Oct 01, 2012 at 15:56 UTC ( #996684=note: print w/replies, xml ) Need Help??

in reply to Comparing sets of phrases stored in a database?

As a slight parenthetical comment to BrowserUK’s excellent (and heavily up-voted) advice, I have also found it useful in some situations to create an additional mathematical set (bit-string ...) which represents “categories” of words that are present in the result.   (The word, “category,” meaning precisely whatever-it-makes-sense-to-you for it to mean.)   The idea is that a phrase could therefore quickly be eliminated from consideration if its set of categories-present is not the same.   (If the category taxonomy contained, say, less than 32 bits, it could be represented as an integer.)   The idea being to reduce the search-space, if say some kind of database I/O is required.

The notion is 150% dependent upon the needs of this application scenario.   First of all, the notion of creating a digest at all must “make sense” in terms of the application spec ... there must be a “natural taxonomy” that the app can usefully exploit “for free.”   Then, it must be shown to called-for, and beneficial.   For instance, in some high-performance critical scenarios, it might well be contra-indicated to store the bit-strings in any sort of database at all:   even a brute-force search of data, if known to entirely fit within the process’s projected resident working-set, avoids all I/O operations entirely.   But if the data were much larger, the notion of creating a lookup digest might apply.

  • Comment on Re: Comparing sets of phrases stored in a database?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://996684]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (9)
As of 2019-11-13 12:12 GMT
Find Nodes?
    Voting Booth?
    Strict and warnings: which comes first?

    Results (72 votes). Check out past polls.