Decoding @ARGV [Was: uparse - Parse Unicode strings]

[jo37]

Hi Ken!

Tried to find a general solution to the problem reported in Re: uparse - Parse Unicode strings.

---

Short explanation of the problem:
There are two basic ways to get correct UNICODE input from the elements in @ARGV:

• implicit decoding with a runtime option -CA or an environment setting PERL_UNICODE=A
• explicit decoding using Encode::decode

Either may be used, but not both.

---

A script that expects UNICODE data from @ARGV cannot easily detect if the implicit decoding is in effect, especially because -CAL makes the behaviour locale-dependent.

The best solution I could find is to check if the data in question is already marked to be in UTF-8. Encode::is_utf8 (or the equivalent utf8::is_utf8) may be used to check this flag, which results in a small modification to your script:

diff --git a/uparse b/uparse
index f5edb92..b05e12a 100755
--- a/uparse
+++ b/uparse
@@ -23,11 +23,11 @@ use constant {
     NO_PRINT => "\N{REPLACEMENT CHARACTER}",
 };
 
-use Encode 'decode';
+use Encode qw(decode is_utf8);
 use Unicode::UCD 'charinfo';
 
 for my $raw_str (@ARGV) {
-    my $str = decode('UTF-8', $raw_str);
+    my $str = is_utf8($raw_str) ? $raw_str : decode('UTF-8', $raw_str
+);
     print "\n", SEP1;
     print "String: '$str'\n";
     print SEP1;

What do you think about this?

[kcott]

++ Thanks for your analysis and patch.

I had planned, assuming I had sufficient time to spare, to have a further look at uparse this weekend and get it to work on different platforms, and in various environments. What you've provided is a good start and helps a lot.

[kcott]

[A follow-up to "Re: Decoding @ARGV [Was: uparse - Parse Unicode strings]".]

I'm not going to have sufficient spare time to do all that I wanted this weekend. I have managed to incorporate your changes and do a couple of other minor things.

When prefixing the uparse command with PERL_UNICODE=A or PERL_UNICODE=SDAL, I just get "Wide character at ..." and no other output. I made these changes:

• Added changes from your patch.
• Changed "use open IO ..." to "use open OUT ...".
• Modified the code layout (mostly to avoid wrapping in PM).

Here's the new code:

#!/usr/bin/env perl

BEGIN {
    if ($] < 5.007003) {
        warn "$0 requires Perl v5.7.3 or later.\n";
        exit;
    }

    unless (@ARGV) {
        warn "Usage: $0 string [string ...]\n";
        exit;
    }
}

use 5.007003;
use strict;
use warnings;
use open OUT => qw{:encoding(UTF-8) :std};
use constant {
    SEP1 => '=' x 60 . "\n",
    SEP2 => '-' x 60 . "\n",
    FMT => "%s\tU+%-6X %s\n",
    NO_PRINT => "\N{REPLACEMENT CHARACTER}",
};

use Encode qw{decode is_utf8};
use Unicode::UCD 'charinfo';

for my $raw_str (@ARGV) {
    my $str = is_utf8($raw_str)
            ? $raw_str
            : decode('UTF-8', $raw_str);
    print "\n", SEP1;
    print "String: '$str'\n";
    print SEP1;

    for my $char (split //, $str) {
        my $code_point = ord $char;
        my $char_info = charinfo($code_point);

        if (! defined $char_info) {
            $char_info->{name}
                = "<unknown> Perl $^V supports Unicode "
                . Unicode::UCD::UnicodeVersion();
        }

        printf FMT, ($char =~ /^\p{Print}$/ ? $char : NO_PRINT),
                    $code_point, $char_info->{name};
    }

    print SEP2;
}

Here's a test run with just uparse:

$ uparse 👮🏼 👮🏼‍♀️ 👮🏼‍♂️

============================================================
String: '👮🏼'
============================================================
👮      U+1F46E  POLICE OFFICER
🏼      U+1F3FC  EMOJI MODIFIER FITZPATRICK TYPE-3
------------------------------------------------------------

============================================================
String: '👮🏼‍♀️'
============================================================
👮      U+1F46E  POLICE OFFICER
🏼      U+1F3FC  EMOJI MODIFIER FITZPATRICK TYPE-3
        U+200D   ZERO WIDTH JOINER
♀       U+2640   FEMALE SIGN
        U+FE0F   VARIATION SELECTOR-16
------------------------------------------------------------

============================================================
String: '👮🏼‍♂️'
============================================================
👮      U+1F46E  POLICE OFFICER
🏼      U+1F3FC  EMOJI MODIFIER FITZPATRICK TYPE-3
        U+200D   ZERO WIDTH JOINER
♂       U+2642   MALE SIGN
        U+FE0F   VARIATION SELECTOR-16
------------------------------------------------------------

And again, this time with PERL_UNICODE=A:

$ PERL_UNICODE=A uparse 👮🏼 👮🏼‍♀️ 👮🏼‍♂️

============================================================
String: '👮🏼'
============================================================
👮      U+1F46E  POLICE OFFICER
🏼      U+1F3FC  EMOJI MODIFIER FITZPATRICK TYPE-3
------------------------------------------------------------

============================================================
String: '👮🏼‍♀️'
============================================================
👮      U+1F46E  POLICE OFFICER
🏼      U+1F3FC  EMOJI MODIFIER FITZPATRICK TYPE-3
        U+200D   ZERO WIDTH JOINER
♀       U+2640   FEMALE SIGN
        U+FE0F   VARIATION SELECTOR-16
------------------------------------------------------------

============================================================
String: '👮🏼‍♂️'
============================================================
👮      U+1F46E  POLICE OFFICER
🏼      U+1F3FC  EMOJI MODIFIER FITZPATRICK TYPE-3
        U+200D   ZERO WIDTH JOINER
♂       U+2642   MALE SIGN
        U+FE0F   VARIATION SELECTOR-16
------------------------------------------------------------

Using "PERL_UNICODE=SDAL" gives the same output as "PERL_UNICODE=A".