Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
The stupid question is the question not asked
 
PerlMonks  

Re: Re: detecting the language of a word?

by tachyon (Chancellor)
on Dec 06, 2002 at 17:33 UTC ( #218106=note: print w/ replies, xml ) Need Help??


in reply to Re: detecting the language of a word?
in thread detecting the language of a word?

There's no way you can decide whether a word is English, German, French, Italian, or whatever other language based on the word alone.

While this statement is true you can take a damn good stab at the language from the words as in plural. In the post below I suggest processing the doc using hash table lookups and counting the instances of putative german, english, french, and italian words. After this is complete you examine the word counts for each languague. As the document length increases so does the probability that the languague with the highest word count is the languague. In fact for docs over a few dozen words you can be almost certain. So you process the document assuming it is german (default lang) and reprocess it using the correct languague as the dominant template if you were wrong.

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print


Comment on Re: Re: detecting the language of a word?
Re: detecting the language of a word?
by Abigail-II (Bishop) on Dec 06, 2002 at 18:17 UTC
    But the goal isn't determining the language of the document. The main language is given - German. The goal is to pick out the individual words that are not German, and mark them accordingly.

    Abigail

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://218106]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (20)
As of 2014-04-16 14:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (428 votes), past polls