Re^10: Encoding Problem - UTF-8 (I think)

And if that was the only unicode encoding, it might be a recommendation; but there are a multitude of "unicode" encodings, the rest of which don't share that property.

Use what works.

That's a bit like saying that a fast poison is better than a slow poison because you suffer less. Basically making a feature of an incidental property that has no value in the real world.

Well, maybe disappear doesn't happen that often... What if they appear instead?

$ touch $'абс\xFF普通话'

$ ls -l

Should software deal with it? What should it do? Let's see

$ echo $'aaa\xFFaaa' | xclip -i # copy to clipboard
[download]

(middle click in the textarea window) aaa�aaa

Looks like Chromium does the right thing...

The world is actually full of garbage strings :)

I'm not even sure that is true -- just move to the end and step backwards

Well, basically, there is a ton of 'false positives'.

$ perl -MEncode -mutf8 -e 'printf "%vx\n", Encode::encode( "UTF-16", "ジ" )'

fe.ff.0.e3.0.82.0.b8
[download]


$ $ perl -MEncode -e 'printf "%vx\n", Encode::decode( "UTF-16", "\xFE\
+xFF\x00\x82\x00\xB8")'
82.b8
[download]

A perfectly good codepoint, unfortunately it's Chinese instead of Japanese...

(it's so painful to make perlmonks display what I want to display... does anyone have some tips? I use <tt> and <p>, it's a pain)

I've never suggesting that we should return to 1-byte encodings; but you have to recognise that variable length encoding undoes 50 years of research into search/sorting/comparison algorithms for no real benefit.

As I said, I see no real benefit in variable length now. Maybe it made some sense when dinosaurs roamed the Earth and modems were 2400 bps.

Very few people (basically, only the US and IETF) went straight from 7-bit to unicode. There are huge amounts of research and data that were produced using JIS/Kanji, Cyrillic, Hebrew, Arabic et al, and unicode is not compatible with any of it.

And any of it is not compatible with each other... so it's not general purpose. Is that unreasonable to expect that a typical computer user (not programmer) in 2015 would be able to use Kanji, Cyrillic, Hebrew, Arabic etc in a single document? (and without pain?) That seems a very reasonable feature request...

No, I don't think it was ever really supposed to make programmers' lives easier. Oh well, c'est la vie.

Comment on Re^10: Encoding Problem - UTF-8 (I think) Select or Download Code

Replies are listed 'Best First'.
Re^11: Encoding Problem - UTF-8 (I think) by Anonymous Monk on Dec 16, 2015 at 18:05 UTC
fe.ff.0.e3.0.82.0.b8 Damn, that's not right. I spent so much time and effort trying to make Perlmonks to show some Unicode that I screwed up (`-mutf8` should've been `-Mutf8`). Anyway, the point was: moving back won't help because we can't find where the error was - almost all combinations of bytes are valid codepoints...	[reply] [d/l] [select]
Re^12: Encoding Problem - UTF-8 (I think) by Anonymous Monk on Dec 16, 2015 at 18:12 UTC
Very few people (basically, only the US and IETF) went straight from 7-bit to unicode. Kind of ironic in this context :)	[reply]


Syntactic Confectionery Delight
	PerlMonks