Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Sorting Vietnamese text

by Anonymous Monk
on Dec 23, 2013 at 06:20 UTC ( #1068139=note: print w/ replies, xml ) Need Help??


in reply to Sorting Vietnamese text

Note: take what I say here with a grain of salt since I know no Vietnamese.

Here's the Vietnamese alphabet sort order. And here's how to read that chart:

  • First column (darkest colour) has the letter in question
  • The other columns have the glyphs that sort under that letter
  • Therefore, ấ and Ầ and ậ sort under (will be found in the dictionary under the heading '')
  • In the case where the two words are otherwise 100% equivalent (except for the diacritics), sort in the left-to-right order given in the chart.

Here's how I handled Japanese sorting (hiragana only) based on a similar chart for Japanese:

sub transliterate {
	my $str = shift;
	$str =~
		tr(がぎぐげござじずぜぞだぢづでどばびぶべぼぱぴぷぺぽっゃゅょ)
		  (かきくけこさしすせそたちつてとはひふへほはひふへほつやゆよ);
	return $str;
}

sub gozyuuon {
	$a->{'sort'} cmp $b->{'sort'} ||
	$a->{'reading'} cmp $b->{'reading'};
}

my @rows = (
	{ word => '同時', reading => 'どうじ' },
	{ word => '当日', reading => 'とうじつ' },
	{ word => '同士', reading => 'どうし' },
	{ word => '投資', reading => 'とうし' },
	{ word => '当時', reading => 'とうじ' },
	{ word => '同室', reading => 'どうしつ' },
);

# create a version with the dakuten (") stripped
for (@rows) {
	$_->{'sort'} = transliterate($_->{reading});
}
for my $row (sort gozyuuon @rows) {

	printf "%s・%s\n", $row->{reading}, $row->{word};
}

Japanese is a bit easier since the unicode codepoints are in correct order already; I only needed to handle the equivalent-sort-order characters.


Comment on Re: Sorting Vietnamese text
Re^2: Sorting Vietnamese text
by Anonymous Monk on Dec 23, 2013 at 07:31 UTC
    And here's what I came up
    sub make_sort_order {
    	my $str = shift;
    	$str =~
    	tr(aảạăaảạăắằẳẵặấầẩẫậbcdđeẻẽẹếềểễệfghiỉĩịjklmnoỏọốồổỗộơớờởỡợpqrstuủũụưứừửữựvwxyỳỷỹỵz)
    	  (00000011111111111112222223456777777888888abcddddddefghijjjjjjkkkkkkllllllmnopqrrrrrrsssssstuvwwwwwwx)d;
    
    	return $str;
    }
    
    my @words = ('ầm', 'm', 'ấm ch', 'm số');
    
    
    print $_->[1], "[n" for
    	sort { $a->[0] cmp $b->[0] || $a->[1] cmp $b->[1] }
    	map  { [ make_sort_order($_), $_ ] } @words;
    

    It's still missing a correct 'secondary sort' (for the edge case when the diacritic-stripped words are identical); it should not be difficult to add once someone figures out a suitable transliteration that sorts asciibetically.

      It's still missing a correct 'secondary sort' (for the edge case when the diacritic-stripped words are identical);

      (laughs) That's hardly an "edge case" in Vietnamese - there are thousands of minimal pairs where the only difference between the words is the diacritical marks. While it's possible to read and understand Vietnamese typed in (7-bit) ASCII without too much ambiguity (i.e. you can figure out what word is meant from the context), this obviously wouldn't work for a dictionary.

      The other issue is that the words in the dictionary need to be sorted in the "correct" order for me to detect duplicates, etc.

      I'll try out your suggestion later today - thanks again!

        I'm probably completely off base here -- why does m số come before m when sorts at the same point as ? And the same question with ỳ ạch and ỷ and the other lines...

        I think there's something wrong with my alphabet -- some characters being eaten perhaps -- since the tr/// has duplicates? If they're wrong, please re-input the correct alphabetical order into the tr/// lines yourself.

        I'm still not sure how spaces play part in all this; I'd sort all the single-character y variants together, and before the two-syllable ones, but the order you've given does not match that and somehow my code handles it handled it even though it shouldn't have.

        You should split the string into two before sorting, though. One part for the actual word, the other for the translation.

        Anyway, here's a try for a correct secondary sort -- of course, I do not have any words to test it on. It sorts how I expect it to do based on what I said in 1068139, but it does not sort how you expect it to.

        Also, why the hell is my tr///d not deleting characters?

        Okay -- the alphabet is only 93 characters per the link in 1068127; you had seven characters extra.

        You should clarify what the relation between monosyllables and polysyllables is, and what is the desired collation behaviour between them.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1068139]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (11)
As of 2014-09-18 13:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (115 votes), past polls