Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re: Re: detecting the language of a word?

by tachyon (Chancellor)
on Dec 06, 2002 at 17:33 UTC ( #218106=note: print w/replies, xml ) Need Help??

in reply to Re: detecting the language of a word?
in thread detecting the language of a word?

There's no way you can decide whether a word is English, German, French, Italian, or whatever other language based on the word alone.

While this statement is true you can take a damn good stab at the language from the words as in plural. In the post below I suggest processing the doc using hash table lookups and counting the instances of putative german, english, french, and italian words. After this is complete you examine the word counts for each languague. As the document length increases so does the probability that the languague with the highest word count is the languague. In fact for docs over a few dozen words you can be almost certain. So you process the document assuming it is german (default lang) and reprocess it using the correct languague as the dominant template if you were wrong.




  • Comment on Re: Re: detecting the language of a word?

Replies are listed 'Best First'.
Re: detecting the language of a word?
by Abigail-II (Bishop) on Dec 06, 2002 at 18:17 UTC
    But the goal isn't determining the language of the document. The main language is given - German. The goal is to pick out the individual words that are not German, and mark them accordingly.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://218106]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (7)
As of 2018-01-16 11:37 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (177 votes). Check out past polls.