Pathologically Eclectic Rubbish Lister | |
PerlMonks |
comment on |
( [id://3333]=superdoc: print w/replies, xml ) | Need Help?? |
Here are some suggestions...
Decouple the article import from detecting duplicates - the latter process could be quite involved, and it will allow you to play with the detection algorithm without interrupting your import process. Have a look at this article: Building a Vector Space Search Engine in Perl. What you need to do could be seen as an instance of the document classification problem. A search engine would help find documents would could be duplicates of a target document, and then you could use something like Text::Similarity to perform a more robust comparison (or flag them for manual review.) In reply to Re: Comparing inputs before inserting into MySQL for "sounds-like" similarity
by pc88mxer
|
|