character substitution in search

[Anonymous Monk]

This is more a program logic question than Perl. I have many documents in a European language site, that users can search using a Perl CGI. The documents are Latin 1, and include special characters such as é, í, etc. Would it be more useful if a user could enter the plain character equivalent and still get results, for example, they enter "geosinteticos" and get documents with "geosintéticos"? Right now they can't. And, I do not know how to begin such a logic, by substituting in the search terms, when a term can contain more than one special character. How is this normally dealt with? I have not found any good examples.

[Ovid]

I am assuming that if your users are going to use your work, they probably have native keyboards (I've tried learning the French one and I can attest that they are a pain). This allows them to type in the alphabet of their choice.

If you wish to capture and use their input in that alphabet, you'll need to learn about using "locale". For example, the following fragment allows Perl to understand French characters.

#!/usr/bin/perl -w

use locale;
use POSIX 'locale_h';

my %locale = (
    French  => "fr_FR.ISO_8859-1",
    English => "us-ascii"
);

setlocale(LC_TYPE, $locale{French}) or die "Invalid locale $locale{Fre
+nch}\n";

You'll need to test this on your system as you may or may not have POSIX support.

The importance of this is being able to compare strings and have the sort order what you expect. Also, it can allow you to deal with upper- and lower-casing characters correctly and handle formatted prints as you expect.

For more information, see Perl locale handling. In the "See Also" section of this link, it lists POSIX(3) sixteen times, so I suspect you want to check that, also :)

As for allowing someone to match characters without accents (resumé|resume), you may wish to check out the String::Approx module from CPAN. Be forewarned, using String::Approx is slow. If performance is a consideration, you may wish to make this function optional for the end user.

Kudos to the Perl Cookbook on this one.

[lhoward]

I have a nasty looking translation statment that "de-accents" Latin 1 characters. I wrote it for indexing and searching web-pages in an accent free way. The important thing to remember is that you must de-accent both the search-term and the text being searched against. Also that this code will only work for the Latin-1 character set.

Here's my tr statment (I'm not using the code tag on purpose 'cause othewise the line will be too long. I've also inserted spaces to help the wrapping... you should remove them if you use this statment):

tr/\xC0\xC1\xC2\xC3\xC4\xC5\xC6\xC7\xC8\xC9\xCA\xCB\xCC\xCD \xCE\xCF\xD0\xD1\xD2\xD3\xD4\xD5\xD6\xD8\xD9\xDA\xDB\xDC \xDD\xDF\xE0\xE1\xE2\xE3\xE4\xE5\xE6\xE7\xE8\xE9\xEA\xEB \xEC\xED\xEE\xEF\xF1\xF2\xF3\xF4\xF5\xF6\xF8\xF9\xFA\xFB \xFC\xFD\xFF/\x41\x41\x41\x41\x41\x41\x41\x43\x45\x45\x45 \x45\x49\x49\x49\x49\x44\x4E\x4F\x4F\x4F\x4F\x4F\x4F\x55 \x55\x55\x55\x59\x73\x61\x61\x61\x61\x61\x61\x61\x63\x65 \x65\x65\x65\x69\x69\x69\x69\x6E\x6F\x6F\x6F\x6F\x6F\x6F \x75\x75\x75\x75\x79\x79/;

[athomason]

Without rules defining what characters to swap out, Perl can't distinguish between characters that should be modified and those that shouldn't. This is a pretty difficult problem, since accent rules alone require splitting words into syllables and knowing the rules for both regular and irregular accentuaton. There are a few solutions here. First, you may indeed want to code all these in, even as rough as that would be. That would entail finding comprehensive rule sets for your language(s) and recording all common exceptions. The opposite way is to decrease the resolution of your search space rather than artificially increase the resolution of your search term (see lhoward's code above). In other words, modify your search indexer so that non-keyboard letters are transliterated into the corresponding keyboard letters. The problem with this, of course, is that you'll get a few false matches. A last approach would be one like killedar was pursuing here, which has the user learn a few non-naturally occuring character combinations which stand in for modified letters. Obviously, users don't like learning how to type again for your site. Then again, there may be a standard way of doing this which I haven't heard of; look around at how other language sites handle the same problem.