Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Re: detecting the language of a word?

by tachyon (Chancellor)
on Dec 06, 2002 at 17:18 UTC ( #218103=note: print w/replies, xml ) Need Help??

in reply to detecting the language of a word?

With 18,000 pages of 300+ words that is at least 3 million words to process. Provided you have the memory by far the fastest thing to do will be to put the word lists into hashes in memory. You would then do something like:

my $german = get_lang_hash('german.txt'); my $english = get_lang_hash('english'); my $french = get_lang_hash('french'); my $italian = get_lang_hash('italian'); my $new_text = ''; for my $word ( split /\b/, $text ) { my $lang = check_word($word); $new_text .= $lang ? qq!<span lang="$lang">word</span>! : $word; } sub check_word { my ($word) = @_; print "got $word\n"; return '' if $german->{$word}; return 'en' if $english->{$word}; return 'fr' if $french->{$word}; return 'il' if $italian->{$word}; return ''; } sub get_lang_hash { my $dict = shift; my %hash; open DICT, $dict or die $!; while (<DICT>) { chomp; $hash{$_}++; } close DICT; return \%hash; }

By splitting on the boundary we will pass punctuation to the check_word() sub but it should not find a match and thus just return ''. The return order from the check word sub detemines our preference. If it could be german we assume it is. If not we see if it could be english, french or italian in that order. If we don't know what it is we call it german and press on.

You should modify this code to count the number of putative german, english, french and italian words in a document. If you find that the english count is >> german then you would reprocess the document with a different check_word() function. In this function you would change the priority order so that english is returned first.... Same for each of the other languagues

You can get an extensive list (250,000) of english words as a flat file word list from The puzzle people seem to have these lists easily and freely available as text files. I presume the same applies for languages other than english.

Any sort of database means disk reads which will be hundreds or thousands of times slower than using an in memory hash table lookup. With memory so cheap and time expensive....

Regardless of what you do you want your word lists to be as complete a possible and do any pre processing before you start on the text.




Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://218103]
[Discipulus]: you are rigth: 2018 had less bad days than 2017, until now!
[Discipulus]: if i had to go to psycho i need to ask why i always end with over complex datastructure: now i have an HoHoAoH in my object
[Discipulus]: i will need to handle by reference later on
[Eily]: kinda sounds like father Christmas is late
[hippo]: (asking because I'm sure you said before about learning English) Did you know there are 2 different words for smaller quantities in English? They are "less" and "fewer". Fewer applies to a number of discrete things like days.
Eily notes also
[hippo]: So if you don't drink as much milk and not as often you are said to drink less milk on fewer days. Confusing, eh?

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (8)
As of 2018-01-23 09:06 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (242 votes). Check out past polls.