in reply to Comparing sets of phrases stored in a database?

As a slight parenthetical comment to BrowserUK’s excellent (and heavily up-voted) advice, I have also found it useful in some situations to create an additional mathematical set (bit-string ...) which represents “categories” of words that are present in the result.   (The word, “category,” meaning precisely whatever-it-makes-sense-to-you for it to mean.)   The idea is that a phrase could therefore quickly be eliminated from consideration if its set of categories-present is not the same.   (If the category taxonomy contained, say, less than 32 bits, it could be represented as an integer.)   The idea being to reduce the search-space, if say some kind of database I/O is required.

The notion is 150% dependent upon the needs of this application scenario.   First of all, the notion of creating a digest at all must “make sense” in terms of the application spec ... there must be a “natural taxonomy” that the app can usefully exploit “for free.”   Then, it must be shown to called-for, and beneficial.   For instance, in some high-performance critical scenarios, it might well be contra-indicated to store the bit-strings in any sort of database at all:   even a brute-force search of data, if known to entirely fit within the process’s projected resident working-set, avoids all I/O operations entirely.   But if the data were much larger, the notion of creating a lookup digest might apply.

  • Comment on Re: Comparing sets of phrases stored in a database?