Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

There's no way you can decide whether a word is English, German, French, Italian, or whatever other language based on the word alone.

While this statement is true you can take a damn good stab at the language from the words as in plural. In the post below I suggest processing the doc using hash table lookups and counting the instances of putative german, english, french, and italian words. After this is complete you examine the word counts for each languague. As the document length increases so does the probability that the languague with the highest word count is the languague. In fact for docs over a few dozen words you can be almost certain. So you process the document assuming it is german (default lang) and reprocess it using the correct languague as the dominant template if you were wrong.

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print


In reply to Re: Re: detecting the language of a word? by tachyon
in thread detecting the language of a word? by domm

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    [robby_dobby]: Corion: Nice, you just won't be around to deal with the mess - enjoy your vacation :-)
    [robby_dobby]: marto: Sure, will do. Thanks!
    [Corion]: marto: No great plans - I'll meet with my sister, my brother and my mother, but that's all :)
    [Corion]: ... and also a visit to my godson+family of course :)
    Corion orders some gifts online :)
    [ambrus]: We're spending Christmas and the week after at my brother's place. First time ever, because the last four years they could came home instead.
    [ambrus]: robby_dobby: wait what? Glasgow? isn't that one of the most boring towns ever?

    How do I use this? | Other CB clients
    Other Users?
    Others cooling their heels in the Monastery: (7)
    As of 2017-12-15 10:41 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      What programming language do you hate the most?




















      Results (431 votes). Check out past polls.

      Notices?