Re: Matching a question in text

[voyager]

Take the text of the email and throw way noise words (what, how, the, etc.). Then take the words left and see if they appear in the faqs. Keep track of how many words in the email match a particular FAQ so you can rank the FAQs.

So when the user clicks "SEND", you can politely say something like "Here is a list of FAQs that might answer your question". Each FAQ has a link to the FAQ and a blurb so user can tell with out going to the FAQ whether it might help.

You should have a button/link in a very obvious place that says, in effect, "None of the FAQs answer my questions, Submit the question Now".

Over time you can tune your list of noise words and perhaps even recognize a list of words that should be given more relevance in sorting the FAQs for display.

[voyager]

Noise words, important words, etc. tend to be domain-specific. What I do for my current project is for every search, I log:
• what was typed (e.g. "show me all the foo and bar")
• what i searched on (e.g., "foo bar") # we use Lingua::Stem and other tricks
• how many "hits"
This is written to a log file and a cron job dumps results into mysql db for easy reporting.

So to finally answer your question, you determine noise words by looking at what your users do. HTH

[swiftone]

Hmm. Interesting. Any ideas for a good source of "noise" words, or do I just fake it?

[mr.nick]

I have a node just for you: Removing Stopwords from a String :)

mr.nick ...

[bastard]

Search engines do this.
It was either htdig or swift-e that had a file that contained such "noise words". Just use that. (I think swift-e had them in it's source code).

You can find links to them here:
http://www.searchtools.com/

On another note, the source is available for alot of the search engines on the page. Code examples for things like fuzzy search and context searching might be available.