Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Proper Unicode handling in Perl

by VK (Novice)
on Sep 02, 2019 at 14:00 UTC ( #11105445=note: print w/replies, xml ) Need Help??

in reply to thread drift is allowed
in thread Is there some universal Unicode+UTF8 switch?

>Thread drift is allowed. For good netiquette, also change the title in the reply form.
OK then. So and first of all I am not a staff developer of Wikipedia, just one of volunteer editors. We needed a script for a set of users willing to get notifications about upcoming internal elections, acting like a daemon (checking every 24 hrs some place and notify if there is something). gives you anything of your choice (Perl, PHP, Python, C#, you name it) in latest stable versions. I don't like Python, have no idea about C#, remember something about Perl - so I did Perl.

This is to make it clear that the list=allusers query has nothing to do with the actual task. It is only to show the exact data format to query and to expect. The full MediaWiki API help is here:

Now... The script has to be able to handle Unicode/UTF-8/whatever literals in the code: so I needed use utf8; It also has to output it in HTML- so I needed binmode STDOUT, ':utf8';
It also has to receive JSON, decode it, slice it, string compare/replace and all other thing - all with Cyrillic in them. I dropped all (en|de)coding things called in this thread unnecessary so came to:


use strict;
use warnings;

use utf8;
use Encode;

use LWP::UserAgent;
use HTTP::Request::Common;
use HTTP::Cookies;

use JSON;

my $browser = LWP::UserAgent->new;

# they ask to use descriptive user-agent - not LWP defaults
# w:ru:User:Bot_of_the_Seven =Участник:Bot_of_the_Seven
$browser->agent('w:ru:User:Bot_of_the_Seven (LWP like Gecko) We come in peace');

# I need cookies exchange enabled for auth
# here is doesn't matter but to give full LWP picture:

# a very few queries can be done by GET - most of MediaWiki require POST
# so I do POST all around rather then remember where GET is allowed or not:
my $response = $browser->request(POST '',
            'format' => 'json',
            'formatversion' => 2,
            'errorformat' => 'bc',
            'action' => 'query',
            'list' => 'allusers',
            'auactiveusers' => 1,
            'aulimit' => 10,
            'aufrom' => 'Б'

my $data = decode_json($response->content);

my $test_scalar = $data->{query}->{allusers}[0]->{name};

my @test_array = @{$data->{query}->{allusers}}[0..2];


sub display_html {

    my @html = (
        '<!DOCTYPE html>',
        '<meta charset="UTF-8">',
        '<title>Мой тест</title>',
        shift // 'Статус  ОК', # soft OR: 0 and empty string accepted
    # to avoid "wide character" warnings:
    binmode STDOUT, ':utf8';
    print "Content-Type: text/html; charset=utf-8\n\n";
    print join("\n", @html);

Is there anything that might go badly wrong concerning Cyrillic in Unicode/UTF-8?

Replies are listed 'Best First'.
Re: Proper Unicode handling in Perl
by haj (Chaplain) on Sep 02, 2019 at 15:13 UTC

    Nice progress! You don't even need the Encode module :)

    This is a pretty straightforward way to deal with Unicode and UTF-8.

    The remaining mentions of UTF-8 in your code have all their justification:

    • use utf8; tells Perl that your source code comes with UTF-8 encoded literals.
    • binmode STDOUT, ':utf8'; makes Perl spit out the strings in @html properly UTF-8 encoded. You can encode any Unicode character in UTF-8, so no problems here.
    • Content-Type: text/html; charset=utf-8 tells the browser that it has to handle the byte stream as UTF-8 and decode the characters accordingly.

    There are two caveats:

    • Obviously, You need to save your source code UTF-8 encoded.
    • You must check whether the JSON data might, in some circumstances, contain characters which have a special meaning in HTML, in particular < and &. This has nothing to do with Unicode, though. I'm adding the relevant stuff to your sub display_html:
      sub display_html {
          use HTML::Entities;
          my $html_encoded = encode_entities(shift, '<>&"');
          my @html = (
              '<!DOCTYPE html>',
              '<meta charset="UTF-8">',
              '<title>Мой тест</title>',
              $html_encoded // 'Статус  ОК', # soft OR: 0 and empty string accepted
          # to avoid "wide character" warnings:
          binmode STDOUT, ':utf8';
          print "Content-Type: text/html; charset=utf-8\n\n";
          print join("\n", @html);

      This thread is getting long, and I have a couple screenfuls of comments, output, and source, which I'll put between readmore tags to save scrollfingers...

      Thanks all for interesting comments,

        Short answer: I can reproduce your strings '╨╨╛╨╣ ╤╨╡╤╤' and '╨╨░╨▒╨║╨╕╨╜╤ ╨╨╕╤╨░╨╕╨╗╤' from the correct ones 'Мой тест' and 'Бабкинъ Михаилъ'. It looks like you print strings which are intended for a UTF-8-enabled browser to a terminal which doesn't understand UTF-8. I guess you are using a Windows terminal with a CP437-compatible (cyrillic) codepage. Please retry your test after entering the command chcp 65001. Long answer follows.

        I wrote:
        Obviously, You need to save your source code UTF-8 encoded.
        You ask:
        Is this a thing? To my understanding, it is the opinion of the software which opens the file as to what its encoding is.

        Yes, it is a thing, which I keep explaining to people who seem to be unaware of the many places where encoding and decoding take place behind the scenes.

        If you see a cyrillic character on your editor screen, then you see a glyph which looks like, say, Б. The Unicode consortium has assigned the codepoint U+0411 and the name CYRILLIC CAPITAL LETTER BE to this character. When such a character is written to a file, then the editor doesn't paint the glyph, nor does it write the codepoint number. Instead, it converts it into a sequence of bytes according to some encoding. In UTF-8, a Б is represented by the (hexadecimal) sequence D091, in Windows Codepage 1251 it is represented by the sequence C1, and in Windows Codepage 866 it is represented by the sequence 81. About 20 years ago, Roman Czyborra collected these and other encodings of the cyrillic alphabets under The Cyrillic Charset Soup.

        Your editor has to chose one of the encodings. It does so according to some system or user preferences, but every editor is different, and some might not even provide decent information about their choice. If you use, for example, Emacs (available on Windows, too), then the buffer's default encoding is displayed on-screen, but you can override it when saving the file. Editors which claim to support Unicode ought to be able to save files under at least one of the UTF encodings. Maybe other monks have current data, but I recall times where Windows editors like Notepad and Notepad++ saved "Unicode" files under UTF-16-BE, which is not UTF-8 and represents a Б by 0411. This looks like the codepoint number. This is no coincidence, but led some software engineers to the wrong conclusion that this is "the Unicode encoding".

        Now what happens if an editor opens an existing file? Where does it derive its opinion from? Well, in general, it can't. The byte C1 could either mean a Б, or an , if the file was meant to be read as Windows Codepage 1252. A sequence D091 renders as Б under Windows Codepage 1252, as Р under Windows Codepage 1251, and as Б under UTF-8. But again, there is a special case: In UTF-16 encodings, there are two possible ways to write 0411 to disk, depending on whether your hardware architecture is "little endian" or "big endian". To distinguish between these two, the standards use the special character Unicode Character 'ZERO WIDTH NO-BREAK SPACE'. A space which doesn't break words and has no width is pretty invisible, so it doesn't do any harm. Little endian systems write this as FEFF, while big endian system swap the bytes and write FFFE. So, whenever a file starts with either FEFF or FFFE, the editor can with some confidence assume that the encoding is UTF-16-LE or UTF-16-BE, respectively. If that invisible space is the first character of a file, it is called a Byte Order Mark, BOM.

        For UTF-8, the BOM is optional and rarely used, some programs don't like it if it is there, and it has the byte sequence EFBBBF. There is no such thing as a BOM for any of the one-byte encodings. If a file does not start with a BOM, you have nothing.

        Similar things happen when the Perl interpreter reads a file. Per default, Perl 5 expects ISO-8859-1 encoding for its source code, which has no BOM. So, if your source code contains the Byte C1 in a literal, then Perl interprets it as the letter , and if it contains the bytes D091 in a literal, then Perl interprets it as a followed by a non-printable character, because 91 maps to a control character in ISO-8859-1.

        To allow human-readable Unicode characters in Perl 5 sources so that you can write Б instead of "\x{411}", the pragma use utf8; was introduced. This, however, requires that the "\x{411}" has been written to disk as the sequence D091 by your editor. I've already said this: There is no pragma for UTF-16 or any other Unicode (or Cyrillic) encoding.

        You wrote:
        To my eye, he has all of the russian on the hook with his data queries; it's just not getting represented correctly on the terminal that Strawberry Perl gives you.

        This is another wrong assumptions. It isn't Strawberry Perl which gives you the terminal, it is the Windows operating system. And - truth hurts - if you spit out UTF-8 encoded strings to a Windows terminal, then it might or might not create the correct glyphs. The terminal is, like your editor, a piece of software which receives a bunch of bytes and tries to create the correct character glyphs for you, according to some encoding. The default encoding of the Windows terminal is not UTF-8, but instead some codepage defined in the regional settings of the operating system (I'm currently on a Linux box, so doing that recherche is up to you). You can learn what codepage is active with the chcp command, and you can also switch your Windows terminal to UTF-8 with the command chcp 65001.

        As other poster hinted - to get right "8-bits Unicode Transformation Format" (commonly known as "UTF-8") output results it is not sufficient to output it right - it is also needed to output to an environment that is able to interpret UTF-8 properly. In 2019 it takes considerable efforts to find an environment that is not able to do it. Yet as usual Matthew rulez - "and whoever seeks, finds" (Mat. 7:8)

        And yes, it is important do not screw up things by doing wrong file savings. Me personally, I perl with Notepad++ and with its option on "Encoding -> UTF-8" plus flag on "apply to open ANSI files". That keeps me safe for my purposes.

        Because as of the year 2019 Unicode/UTF-8 is not yet another encoding to deal with. It is the encoding, the only one to deal with. Anything else - only for very special occasions when the soul is really asking for adventures. :-)

        (update) With that coming back to my original question. Just try to replace rather convoluted POST calls with LWP native ->get() and ->post() methods - and we are back to the circle one with all response like ‘˜‘›˜žТ•šАР and crap. So LWP gets the right response headers (that it is UTF-8). Program says use utf8. Print says output utf8. Yet everything seems to be staying on the logic "if there is a slightest possibility it is not that damn utf-8, then don't use it; if there is not such possibility, still don't use it". This is what I said in my OP: "in order to achieve it I had to make my script like a drunk buddy I have to get from the bar back home :-) - once I lower attention, he tries to fell on the ground and sleep". It may be funny first but it gets highly irritating soon.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11105445]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2019-12-12 18:21 GMT
Find Nodes?
    Voting Booth?

    No recent polls found