Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
You answered like i suppress unicode to everyone and to everywhere. That's not my goal.

No, and I did not understand you like that. I just wanted to explain why Unicode support still sucks so much. It's not just a Perl problem, some big problems with Unicode are outside our control. I would be happy if I could use utf8_for_everything;, but that cannot work today. We could end implementing use utf8_where_possible; plus the same manual fiddling to turn on Unicode in subsystems that do not (yet) understand use utf8_where_possible; or that need workarounds.

I'd like to have a sandbox in Perl, where unicode were treated naturally.

Unicode is largely treated naturally in Perl (at least since 5.8.1). You put binary data, a string with legacy encoding, or a string with a Unicode encoding into a scalar and everything works as expected. All the magic happens behind the scenes. Things become ugly as soon as you start interfacing with the outside world, e.g. STDIN, STDOUT, STDERR, %ENV, @ARGV, and external libraries (XS code). DBI and the DBD::* modules currently gain more and more Unicode support, simply by reviewing and changing every place in the code where Perl scalars become C strings and vice versa to respect or to set the internal Unicode flag of the Perl scalar. Sometimes by passing a Perl scalar further down into the code instead of passing a C string (this happened with some internal DBI APIs). Sometimes by converting Perls idea of Unicode to and from what the operating system or a library expects. (This happens in DBD::ODBC.)

Trying make it more clear. I am not familiar with Perl history so good, but let me make assumption, that in some phase there was no strict-pragma. OK? Then someone thought, it may be good idea and found ways to implement it. Did that break any earlier code? I don't think so. But it made available widely use strict pragma.

This is part of your problem understanding the problems of Unicode support. Perl has a long history and culture of NOT breaking old code. use strict is an example for this. The inventors of strict could have turned strict on by default, and force people to update the legacy code by either adding no strict or by cleaning up the old junk. This would perhaps reduced the ammount of bad Perl code a lot, and would have forced newbies to write cleaner code. But many people would have gotten very angry because millions lines of code would stop working from one day to the other, just because the lastest f*** Perl update started bean counting instead of getting the job done.

The same thing happened with Unicode support, and you will find some good explainations inside the Perl documentation why Unicode support is largely OFF by default. Turning it on by default would have broken even more code that assumes a character is a byte.

So i am talking now. As far as i see, for module authors is there no possibility to see, do the module caller uses utf8 or not. Am i correct? And, does it break any earlier code, if they would have such a possibility? That would be a single step, IMHO :)

Wrong problem. The module caller may use a mix of Unicode, legacy encoding and binary data at any time. For any function or method in a module or class, it is completely irrelevant if "the caller uses utf8" or not.

Modules (or better: their authors) must no longer assume that scalars contain bytes, they contain arbitary large characters. length returns the number of characters in a scalar. If the internal Unicode flag on a scalar is turned off, the module may safely assume that the scalar contains bytes, either binary data or a legacy encoding. When it is on, it must correctly handle large characters. When interfacing with the outside world (O/S, network, database, ...), it may be necessary to convert the large characters to a different encoding (and back, of course). Whenever scalars are returned, they may either have the Unicode flag set and may contain large characters, or they have the flag cleared and must not contain large characters, not even as a UTF byte stream. (Except, of course, the purpose of the function is to generate UTF byte streams.)

Many modules do not need changes, because they did not assume byte==character from the beginning, and so Perl automatically does the right thing. Some modules tried to handle Unicode all by themselves even before Perl hat Unicode support, Template::Toolkit seems to be such a module. They mostly work, and as long as you don't mix them with really Unicode-capable modules, nothing wrong happens. Their only problem is that their scalars contain UTF byte streams instead of large characters. This can only be solved by either dropping support for legacy Perl versions (i.e. use 5.008_001) or by having the module code behave different for old and new Perl versions.

I have not deeply investigated, how unicode-proof is Linux for now, but on system level i have'nt any complains already years (Debian and Kubuntu). If you could me give some hints, how determine unicode use, i'd like to test it.

OK, some simple problems. "Unicode string" here means a string containing characters outside the ASCII and ISO range, e.g. the smiling face, cyrillic letters, or the like. See for examples. "Legacy string" means a string any non-Unicode encoding, like ASCII, ISO-8859-x, the various asian encodings, and so on.

  • Create two environment variables named FOO and BAR, one with a Unicode string of exactly 10 characters as value, the other one with a legacy string of exactly 10 characters. Choose randomly which variable gets the Unicode string. fork() and exec() some child processes (Scripts in bash, perl, ash, ksh, python, ruby, lua, ..., and perhaps some compiled programs written in C, C++, Assembler, Fortran, ...) and let each process report the number of characters in both FOO and BAR, without(!) telling the child processes which of the two variables actually contains Unicode characters.
  • Create a Perl script that writes randomly either a legacy string or a Unicode string to STDOUT, both containing exactly 10 characters. You may use binmode STDOUT,':utf8' and the like to get rid of all warnings and errors. Create a second program (in Perl or any other language) that reads its STDIN and reports the number of characters it read from STDIN. Connect both programs using a pipe, like this: perl | perl
  • Create a Perl script that randomly selects either a legacy string or a Unicode string of exactly 10 characters and passes that string as only argument to child processes written in various langaues. Each child process must report the number of characters passed as arguments.
  • Create a file whose name is a Unicode string. Does ls display it correctly? On both the console and via telnet and ssh from different other operating systems? Can rm remove it without resorting to rm -rf *? Can you copy and move it inside midnight commander? Does your preferred X environment display the name correctly? On the desktop and in the file manager? Even inside File-Open dialogs? Even in programs that are not part of the X environment (like Firefox)? What about other X environments (Gnome, KDE, xfce, ...)? Can you pass the file as a command line argument to arbitary programs and can they open it? Does the filename still look ok when you share the file via FTP, HTTP, SMB, NFS, rsync? Can you still open it over the network from Linux, Windows, *BSD, Solaris, MacOS? Can you overwrite it over the network?

Yes, this are stupid little tests, except for the last one. They are very similar to what the UnicodeRoundTrip test linked above does. You would not believe how often that simple test broke. And before I added the test, I had even more problems with data that was modified somewhere between Perl and the database engines, causing other tests to break very misteriously or even to fail silently.


Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

In reply to Re^9: any use of 'use locale'? (source encoding) by afoken
in thread any use of 'use locale'? by wanradt

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others pondering the Monastery: (5)
    As of 2018-05-24 00:55 GMT
    Find Nodes?
      Voting Booth?