|Perl: the Markov chain saw|
Re: OT: Vector based search engineby DrHyde (Prior)
|on May 14, 2004 at 09:13 UTC||Need Help??|
I implemented this at work. Sadly, I can't (yet) share the code, but I can certainly explain how it works.
As the article explains, you grovel over your corpus and extract a list of all the unique words from that corpus (after stemming and removing stopwords, of course). Each of those words will be represented by a dimension in an N-dimensional space. So, for example, if your list of words is "beer", "pie", "ninja" (the three best things ever), then you will be working with a three dimensional space whose axes are labelled "beer", "pie" and "ninja" instead of x, y and z.
Once you have the list of words, you construct a vector in your N-dimensional space for each document. You simply go through each document, stemming all the words and removing stopwords again, and each time a word appears, you add one to the relevant dimension.
The document "ninjas like pie" would therefore become:
[0, 1, 1]
and the document "I'll have beer, beer, more beer and a pie please" would be:
[3, 1, 0]
Those look exactly like co-ordinates in the N-dimensional space, which they are. However, because of what we do with them later, it is better to think of them as vectors - that is, they are directions from the [0, 0, 0] point, the "origin". [0, 1, 1] represents the direction "due east and up at an angle of 45 degrees", for example.
So now we have reduced each document to a vector, which we can store in a nice compact index. To search the index, we take the user's search terms and reduce them to a vector just like we did with the documents. Any words which don't appear in the index - ie for which we don't have a dimension - are ignored.
Now we need to see how close this vector is to all of the vectors in the index. Seeing that all the vectors are lines which cross at the origin, we can define their closeness as the angle between them. The smaller the angle, the closer the vectors. You need to do this for *all* document vectors.
You will note that there is an angle between all possible vectors, so you need to choose a cut-off angle so that you only return results which are reasonably close. What that angle is will depend on your data and your usage patterns.
The advantages of a vector search are:
The disadvantages are: