Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Re: Language translator and dictionaries

by elef (Friar)
on Mar 19, 2011 at 16:26 UTC ( #894162=note: print w/replies, xml ) Need Help??

in reply to Language translator and dictionaries

It's pretty easy to get dictionaries for these languages online, so if you want to build a perl app that "translates" word by word, that would be reasonably easy to do.
If you attempt to do this, the quality of the output will be very poor. It should work passably well if the input data is word lists (especially if they cover one given area and you can find good glossaries for that area) but if your input data is running text of any complexity, the "translation" will be barely intelligible at best.

Some sort of MT (machine translation) engine would do a much better job, and I'm sure there are some rudimentary open source solutions out there, but I don't know about the specifics. If you have money to spend, there are more than a few companies happy to sell you a complete system.

How to get dictionaries:
I'm basing this on the assumption that you'll need to handle EU languages only.
- Get multilingual dictionaries/word lists. These will cover the most basic words.
- Extract term pairs from Wikipedia dumps. The xml dumps contain links to the same article in other languages, so you can extract well north of 100,000 term pairs for major combinations like French-English. I can give you a brief howto if needed.
- Same with Wiktionary.
- Grab EU term lists such as Eurovoc and the CPV. They contain thousands of terms in all the EU languages.
- If you need multilingual text corpora for training a MT system, grab the DGT-TM and the europarl corpus. These contain about a million sentences each of sentence-aligned text in EU languages. You can also add UN texts, but the UN only has 6 official languages.

If you go the dictionary route, you should probably run the text through a stemmer. Open source toolchain for tasks like this:

Hunalign (see link) builds crude dictionaries as a by-product of text alignment, so in principle you could let it loose on the europarl and DGT-TM texts and have it produce dictionaries to complement whatever you find online. It may not be good enough quality to be worth your time, though.
  • Comment on Re: Language translator and dictionaries

Replies are listed 'Best First'.
Re^2: Language translator and dictionaries
by vit (Friar) on Mar 19, 2011 at 16:48 UTC
    Thanks a lot, Elef, for your detailed response. I will explain you my purpose so that you might have an idea what could be the best for me.
    I know a little about machine translation stuff but it's not what I need for my project. I am working on the text classification which is using a bag-of-words approach. So my translation task is very easy. Just word-by-word since any text will be split into words anyway. And all EU languages into English only. Which also simplifies the task.
    I think using a third party translation servers is an overhead to me and it is not very reliable especially when it comes to massive queries. So my guess is that I should use some simple word-to-word translation procedure on my server.
      word-by-word since any text will be split into words anyway. And all EU languages into English only

      Well, then you should probably start building your multilingual dictionary. I don't think Google Translate would be too happy about you making thousands of automated single-word queries.

      Other EU term lists:

      Add whatever you can extract from Wikipedia and Wiktionary dumps and you should be set.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://894162]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (11)
As of 2018-06-22 15:32 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (124 votes). Check out past polls.