JSON, UTF-8 and Filehandles

[Kirsle]

The JSON module supports UTF-8 encoding, but there's a peculiar quirk when writing directly to a file that was opened in UTF-8 mode.

Backstory: I have a Perl site that stores all its "database" tables as flat JSON documents on disk. I wanted to make sure Unicode is supported all throughout the system, so I was running some experiments to make sure I'm reading/writing to the files properly.

Here is a script I have for testing:

#!/usr/bin/perl -w

use 5.14.0;
use utf8;
use Encode;
use JSON;

my $json = JSON->new->utf8->pretty();

STDOUT->binmode(":utf8");
my $umbreon = "ブラッキー";
my $test    = "\x{100}\x{2764}";

say "Name: $umbreon (is utf8: " . utf8::is_utf8($umbreon) . ")";
say "Test: $test (is utf8: " . utf8::is_utf8($test) . ")";

my $data = {
   name => $umbreon,
   test => $test,
};

open (my $fh, ">", "utf8-test.txt");
binmode($fh, ":utf8");
print {$fh} "Name: $umbreon\nTest: $test\n";
close($fh);

my $encoded = $json->encode($data);
print "encoded (original): $encoded (is utf8: " . utf8::is_utf8($encod
+ed) . ")\n";
$encoded = Encode::decode("utf-8", $encoded);
print "encoded: $encoded (utf8: " . utf8::is_utf8($encoded) . ")\n";
open (my $fh2, ">", "utf8-test.json");
binmode($fh2, ":utf8");
print {$fh2} $encoded;
close($fh2);
__END__

[kirsle@vostro ~]$ perl utf8-test.pl 
Name: ブラッキー (is utf8: 1)
Test: Ā❤ (is utf8: 1)
encoded (original): {
   "test" : "�€❤",
   "name" : "�ƒ–�ƒ��ƒƒ�‚��ƒ�"
}
 (is utf8: )
encoded: {
   "test" : "Ā❤",
   "name" : "ブラッキー"
}
 (utf8: 1)

From tinkering with this code, some observations:

• The "use utf8" is absolutely crucial. Otherwise, applying utf8 encoding to either STDOUT or the filehandle would mangle the text.
• If you don't "use utf8", and also don't change the binmodes, the Unicode text still prints intact to both the terminal and files. What gives? Does it inherit the system encoding or something?
• My JSON object is initialized with its utf8 option.
• $json->encode corrupts the UTF-8 (if printed to either place, the UTF-8 strings are corrupted and the is_utf8 flag is undef).
• When I implemented this on my site, I found that I had to:
  • When writing JSON, I had to call JSON's encode() first and then UTF-8 decode the output before writing to disk.
  • When reading from disk, I had to UTF-8 encode (mangle) the text from disk before giving it to JSON's decode(), otherwise JSON croaks about wide characters.

Imho, UTF-8 in Perl is full of confusing quirks. For my fellow monks, here is a pair of functions I wrote that will recursively switch on (or off) the UTF-8 flag on a data structure in Perl:

=head2 data utf8_decode (data)

Recursively UTF8-decode a data structure. Any data structure.

Decoding turns on the UTF-8 flag in Perl and makes Perl treat the data
+ as
string (so methods like C<length()> are accurate). If you want to prin
+t the
data over the network, you need to B<encode> it into bytes.

=cut

sub utf8_decode {
    my $data = shift;
    my $encode = shift || 0;

    # If it's a data structure (hash or array), recurse over its conte
+nts.
    if (ref($data) eq "HASH") {
        foreach my $key (keys %{$data}) {
            # Another data structure?
            if (ref($data->{$key})) {
                # Recurse.
                $data->{$key} = utf8_decode($data->{$key}, $encode);
                next;
            }

            # Encode the scalar.
            $data->{$key} = utf8_decode($data->{$key}, $encode);
        }
    }
    elsif (ref($data) eq "ARRAY") {
        foreach my $key (@{$data}) {
            # Another data structure?
            if (ref($key)) {
                # Recurse.
                $key = utf8_decode($key, $encode);
                next;
            }

            # Encode the scalar.
            $key = utf8_decode($key, $encode);
        }
    }
    else {
        # This is a leaf node in our data (a scalar). Encode UTF-8!
        my $is_utf8 = utf8::is_utf8($data);

        # Are they *encoding* (turning into bytes) instead of *decodin
+g*?
        if ($encode) {
            # Encoding (making bytestream): only decode IF it is curre
+ntly UTF8.
            return $data unless $is_utf8;
            $data = Encode::encode("UTF-8", $data);
        }
        else {
            # Decoding. If it's ALREADY UTF-8, do not decode it again.
            return $data if $is_utf8;
            $data = Encode::decode("UTF-8", $data);
        }
    }

    return $data;
}

=head2 data utf8_encode (data)

Recursively UTF8 encode a data structure. B<Encoding> means turning th
+e data
into a byte stream (so string operators like C<length()> will be inacc
+urate).

Encoding is necessary to transmit a Unicode string over a network.

=cut

sub utf8_encode {
    my $data = shift;
    return utf8_decode($data, 1);
}

This is handy when you need to convert strings to or from byte streams. I apply a decode to all the incoming CGI params (just to be sure they're all UTF-8), and when I need to take an md5_hex of a string, I encode it to bytes first (because Digest::MD5 will croak if you try to md5_hex a string with wide characters).

[juster]

is_utf8 is used to check if perl's internal representation of a string has been "upgraded" to utf8. That is, the string cannot possibly be represented in ascii, such as your: my $test = "\x{100}\x{2764}";. When you decode a utf8 string it is converted to perl's internal representation. This just happens to be utf8, or something close, but it doesn't have to be and worrying about it is just a waste of time.

Really, you only have to worry about decoding input and encoding output (edit: I had encoding/decoding backwards!). You received mangled text on your files/screen because you are having JSON encode it's results as a utf8-encoded string. JSON returns a string of bytes that are utf8 encoded.

Next, when you write to the filehandle, having set :utf8 on the filehandle, your bytes are automatically encoded to utf8 again. You have doubly-encoded the string and it is quite predictably garbage.

Quoth the utf8 manpage:

Do not use this pragma for anything else than telling Perl that your script is written in UTF-8.

Read the links under the SEE ALSO section of the utf8 docs for more about unicode. I also had this bookmarked, I remember it being pretty good: http://www.ahinea.com/en/tech/perl-unicode-struggle.html

[Kirsle]

Aha, thanks.

I went and took a second look at the JSON manpage too.. apparently the utf8 option in JSON already takes the liberty of encoding the output (removing the UTF-8 flag in Perl, making it render as garbage), which is good if you're using JSON.pm to send/receive data over a network socket. Without the utf8 option, JSON still "supports" UTF-8, it just doesn't automatically do the string encoding/decoding.

I find it a little bit weird that, if you don't use utf8;, and open a filehandle in UTF-8 mode, that wide characters printed to it are mangled; is this because Perl doesn't expect Unicode to be written to the filehandle (because the script printing it didn't use utf8), so it double-encodes it?

[juster]

I wrote the first reply in about 10 minutes and had to leave. Let me be more clear about the is_utf8 flag check. Don't use is_utf8. is_utf8 checks if a string is internally encoded in utf8. Deep inside the angry bowels of perl! Using is_utf8 is fraught with peril, which is unfortunate for such a seemingly easy function, right? It doesn't do what you think it does.

You didn't read the unicode docs did you? Here is a great link: http://perldoc.perl.org/perlunifaq.html#What-is-%22the-UTF8-flag%22%3f. There is also perluniintro, perlunicode, utf8 etc. Feel free to continue screwing yourself by not reading these. Don't forget to not read the link I gave in my first reply.

Now I have time to reply to your bullets:

• use utf8 is necessary for writing your source code in utf8. This is only useful for writing string literals in utf8, since there is not yet a snowman operator (perl6?). Your output is probably garbled because your string literal ($umbreon) is written in utf8 and perl has no way to know this without the use utf8.
• Your terminal/shell is utf8 compatible. It translates everything as utf8. You can print each byte of a utf8 character separately and your terminal would decode it as utf8.
• The UTF8 is not corrupted. The UTF8 is just fine. You are encoding it twice.
• • You are encoding, decoding, and encoding again.
  • With utf8 turned on, JSON will decode the byte string you provide it from utf8 to perl's internal string representation.

In response to your latest reply: Stop worrying about the utf8 flag and just worry about encoding once and decoding once. Don't encode with JSON if you are encoding to utf8 before writing to the file. Vice-versa with decode. That's all you need to worry about. Remember, this also applies to STDOUT.

The wide characters are probably mangled because you are using a utf8 string constant.