Perl: the Markov chain saw | |
PerlMonks |
Re: De Duping Street Addresses Fuzzilyby legato (Monk) |
on Feb 01, 2005 at 04:06 UTC ( [id://426802]=note: print w/replies, xml ) | Need Help?? |
Well, the first thing I would do is use the US Postal Service Web API. You need to request permission, but they almost always grant it (there's even a checkbox on the application that says you'll be using it to cleanse address databases). This will allow you to send the addresses as they exist in your DB, and get the USPS normalized "official" address back. For example: Might give you:
Variations on that address should result in the same canonical address. You can then compare to see if there are duplicate canonical addresses with a simple string equality. You may need to do a little data cleaning, like removing multiple spaces ($address=~s{\s+}{\x20}g;, for example) before running the compare, but this should catch the vast majority of your duplicates. Anima Legato
In Section
Seekers of Perl Wisdom
|
|