in reply to Language translator and dictionaries
It's pretty easy to get dictionaries for these languages online, so if you want to build a perl app that "translates" word by word, that would be reasonably easy to do.
If you attempt to do this, the quality of the output will be very poor. It should work passably well if the input data is word lists (especially if they cover one given area and you can find good glossaries for that area) but if your input data is running text of any complexity, the "translation" will be barely intelligible at best.
Some sort of MT (machine translation) engine would do a much better job, and I'm sure there are some rudimentary open source solutions out there, but I don't know about the specifics. If you have money to spend, there are more than a few companies happy to sell you a complete system.
How to get dictionaries:
I'm basing this on the assumption that you'll need to handle EU languages only.
- Get multilingual dictionaries/word lists. These will cover the most basic words.
- Extract term pairs from Wikipedia dumps. The xml dumps contain links to the same article in other languages, so you can extract well north of 100,000 term pairs for major combinations like French-English. I can give you a brief howto if needed.
- Same with Wiktionary.
- Grab EU term lists such as Eurovoc and the CPV. They contain thousands of terms in all the EU languages.
- If you need multilingual text corpora for training a MT system, grab the DGT-TM and the europarl corpus. These contain about a million sentences each of sentence-aligned text in EU languages. You can also add UN texts, but the UN only has 6 official languages.
If you go the dictionary route, you should probably run the text through a stemmer. Open source toolchain for tasks like this: http://mokk.bme.hu/resources/
Hunalign (see link) builds crude dictionaries as a by-product of text alignment, so in principle you could let it loose on the europarl and DGT-TM texts and have it produce dictionaries to complement whatever you find online. It may not be good enough quality to be worth your time, though.
If you attempt to do this, the quality of the output will be very poor. It should work passably well if the input data is word lists (especially if they cover one given area and you can find good glossaries for that area) but if your input data is running text of any complexity, the "translation" will be barely intelligible at best.
Some sort of MT (machine translation) engine would do a much better job, and I'm sure there are some rudimentary open source solutions out there, but I don't know about the specifics. If you have money to spend, there are more than a few companies happy to sell you a complete system.
How to get dictionaries:
I'm basing this on the assumption that you'll need to handle EU languages only.
- Get multilingual dictionaries/word lists. These will cover the most basic words.
- Extract term pairs from Wikipedia dumps. The xml dumps contain links to the same article in other languages, so you can extract well north of 100,000 term pairs for major combinations like French-English. I can give you a brief howto if needed.
- Same with Wiktionary.
- Grab EU term lists such as Eurovoc and the CPV. They contain thousands of terms in all the EU languages.
- If you need multilingual text corpora for training a MT system, grab the DGT-TM and the europarl corpus. These contain about a million sentences each of sentence-aligned text in EU languages. You can also add UN texts, but the UN only has 6 official languages.
If you go the dictionary route, you should probably run the text through a stemmer. Open source toolchain for tasks like this: http://mokk.bme.hu/resources/
Hunalign (see link) builds crude dictionaries as a by-product of text alignment, so in principle you could let it loose on the europarl and DGT-TM texts and have it produce dictionaries to complement whatever you find online. It may not be good enough quality to be worth your time, though.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: Language translator and dictionaries
by vit (Friar) on Mar 19, 2011 at 16:48 UTC | |
by elef (Friar) on Mar 19, 2011 at 17:36 UTC |
In Section
Seekers of Perl Wisdom