Beefy Boxes and Bandwidth Generously Provided by pair Networks BBQ
Perl Monk, Perl Meditation
 
PerlMonks  

Deciding which word in an array is the closest match to a given word

by bilbo800 (Initiate)
on Sep 09, 2004 at 09:32 UTC ( #389603=perlquestion: print w/ replies, xml ) Need Help??
bilbo800 has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks

A classic html scraping problem, but could not find a spot on solution.

I have a list of products which I collect online prices for from different on line sites. Problem is on some sites the product is spelled ‘sony dhc15E’, on some it’s just ‘dhc15-E’ and on some it’s ‘Sony camcoder 15-e’.

Assuming I have a list of 200 such products, how do I approach the problem of taking the product name from the merchant site and finding the best suitable match in my array of product.

Thx

Oron

Comment on Deciding which word in an array is the closest match to a given word
Re: Deciding which word in an array is the closest match to a given word
by eserte (Deacon) on Sep 09, 2004 at 09:51 UTC
    Try either String::Approx or String::Similarity. The both modules are different in their approach: the former returns all matches based on an "error" or "fuzziness" parameter while the latter returns a similarity factor of two strings. Both can be tailored for your needs.
      No, that won't work because the modules are too general. "dhc15-E" and "Sony camcoder 15-e" should be a match, but something like "dhc16-E" should not match, as that will be a camcorder of a different type. But the modules you mention won't have the knowledge what the strings mean, and will consider "dhc15-E" and "dhc16-E" quite similar - as they differ by only one character.
        This looks like you need to put some custom logic in it. From your example it looks like the substring "dhc" might be substituted with "Sony camcorder". Maybe you can try to use a number of such mappings to get a canonical form. I can also imagine that dashes and spacing may differ, so strip all non-characters before comparing.
      This node was taken out by the NodeReaper on Thu Sep 9 09:24:33 2004 (EST)
      Reason: grinder delete: back-arrow submit dup of id://389619

      For more information on this node visit: this

Re: Deciding which word in an array is the closest match to a given word
by wfsp (Abbot) on Sep 09, 2004 at 09:57 UTC
    Do you know all the variations in advance? If you do I would suggest a look up table (hash). The keys would be all the variations you would expect and the value would be whatever is in your array.

Re: Deciding which word in an array is the closest match to a given word
by Random_Walk (Parson) on Sep 09, 2004 at 10:52 UTC
    Perhaps you need a mixture of the two suggested approaches. When the best possible match is found by one of the string aproximation packages comparing to all matches already known you asign it as a best guess match. You also add this match to a list for human review of strings that were matched and the cannonical product name. Once a human reviewer agrees a match is good it goes into the hash of know matches

    You will never get 100% as some very different products may be given the same name (e.g. an F15 could be an aircraft or a sunscreen)

    Cheers,
    R.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://389603]
Approved by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (4)
As of 2014-04-19 23:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (485 votes), past polls