Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: detecting the language of a word?

by Abigail-II (Bishop)
on Dec 06, 2002 at 16:56 UTC ( #218098=note: print w/ replies, xml ) Need Help??


in reply to detecting the language of a word?

There's no way you can decide whether a word is English, German, French, Italian, or whatever other language based on the word alone. Think about it - if it was that simple, there wouldn't be reason to markup the words to aid the speech sofware - the software would already know.

You can't use a simple lookup in a database, there are words in English that exists in other language as well - which sometimes a totally different meaning. If you are going to automate this (and this isn't an easy task, people have been working on this for decades), you will have to look at the context. You will have to parse the sentences, and actually understand each word. And even then you might have problems with sentences like 'time flies like an arrow'.

I suggest that you research the literature on computarized linguistics, automatic translations, and speech software.

Abigail


Comment on Re: detecting the language of a word?
Re: Re: detecting the language of a word?
by tachyon (Chancellor) on Dec 06, 2002 at 17:33 UTC

    There's no way you can decide whether a word is English, German, French, Italian, or whatever other language based on the word alone.

    While this statement is true you can take a damn good stab at the language from the words as in plural. In the post below I suggest processing the doc using hash table lookups and counting the instances of putative german, english, french, and italian words. After this is complete you examine the word counts for each languague. As the document length increases so does the probability that the languague with the highest word count is the languague. In fact for docs over a few dozen words you can be almost certain. So you process the document assuming it is german (default lang) and reprocess it using the correct languague as the dominant template if you were wrong.

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

      But the goal isn't determining the language of the document. The main language is given - German. The goal is to pick out the individual words that are not German, and mark them accordingly.

      Abigail

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://218098]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (6)
As of 2014-12-20 03:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (95 votes), past polls