Unknown charnames when building Encode

yulivee07 has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow Perlmonks, I am trying to build Encode 2.88-3 from CPAN on AIX 7.2. During the make test phase I receive various errors about unknown characters:

Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per
+l5/lib/5.20.1/_charnames.pm line 499.
Unknown charname 'alpha' at t/Encode.t line 44, within string
BEGIN not safe after errors--compilation aborted at t/Encode.t line 14
+8.
t/Encode.t ................. 
t/encoding-locale.t ........ ok
Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per
+l5/lib/5.20.1/_charnames.pm line 459.
Unknown charname 'LATIN SMALL LETTER SHARP S' at t/encoding.t line 77,
+ within string
BEGIN not safe after errors--compilation aborted at t/encoding.t line 
+165.
Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per
+l5/lib/5.20.1/_charnames.pm line 459.
Unknown charname 'POUND SIGN' at t/mime-header.t line 166, within stri
+ng
Execution of t/mime-header.t aborted due to compilation errors.
# Looks like your test exited with 2 just after 1.
t/mime-header.t ............
[download]

To test whether this is an AIX problem or a perl problem, I tried to build the same version on my linux system, where Encode installs just fine.
To pick the first error:

is "\N{alpha}",substr($uni,0,1),"alpha does not map to symbol 'a'";
[download]

It seems AIX perl is unable to find the \N{alpha} character. I am a bit lost here - where does perl usually search for characters like this?
I need a hint into the direction I have to search for with this problem. Can someone provide some debugging tips?
Updates:

perl -E 'use charnames (); say $charnames::VERSION'
1.40

perl -C -E 'say "\x{3b1} - \x{df} - \x{a3}"'
Wide character in say at -e line 1.
&#945; - ß - Ł

perl -E 'use Unicode::UCD; say $Unicode::UCD::VERSION'  
0.58

perl -MUnicode::UCD=charprops_all -E 'say charprops_all("U+$_")->{Age}
+ for qw{3b1 df a3}'
"charprops_all" is not exported by the Unicode::UCD module
Can't continue after import errors at -e line 0.

perl -C -E 'say "\N{greek:alpha}"'
Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per
+l5/lib/5.20.1/_charnames.pm line 459.
Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per
+l5/lib/5.20.1/_charnames.pm line 499.
Unknown charname 'greek:alpha' at -e line 1, within string
Execution of -e aborted due to compilation errors.
[download]

The build process is using CPAN (perl -MCPAN -eshell) to install modules. We use local::lib to install to a specific directory, rather than the system perl path. The perl we are using is the one coming with AIX 7.2, so we did not build perl ourself.

perl -V
Summary of my perl5 (revision 5 version 20 subversion 1) configuration
+:
   
  Platform:
    osname=aix, osvers=6.1.0.0, archname=aix-thread-multi
    uname='aix blade08 1 6 00003c3ad100 '
    config_args='-d -Dprefix=/usr/opt/perl5 -Dcc=xlc_r -Duseshrplib -D
+usethreads'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='xlc_r -q32', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX
+_SOURCE -qmaxmem=-1 -qnoansialias -qlanglvl=extc99 -DUSE_NATIVE_DLOPE
+N -DNEED_PTHREAD_INIT -q32 -D_LARGE_FILES',
    optimize='-O',
    cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=
+-1 -qnoansialias -qlanglvl=extc99 -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_
+INIT'
    ccversion='12.1.0.9', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='ld', ldflags =' -brtl -bdynamic -b32'
    libpth=/lib /usr/lib /usr/ccs/lib
    libs=-lbind -lnsl -ldbm -ldl -lld -lm -lcrypt -lpthreads -lc
    perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc
    libc=, so=a, useshrplib=true, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -bE:/usr/
+opt/perl5/lib/5.20.1/aix-thread-multi/CORE/perl.exp'
    cccdlflags=' ', lddlflags='-bhalt:4 -G -bI:$(PERL_INC)/perl.exp -b
+E:$(BASEEXT).exp -bnoentry -lpthreads -lc -lm'


Characteristics of this binary (from libperl): 
  Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
                        USE_ITHREADS USE_LARGE_FILES USE_LOCALE
                        USE_LOCALE_COLLATE USE_LOCALE_CTYPE
                        USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
                        USE_REENTRANT_API
  Built under aix
  Compiled at Feb  6 2015 14:54:29
  %ENV:
    PERL5LIB="/home/perl_ss/perl5/lib/perl5/aix-thread-multi:/home/per
+l_ss/perl5/lib/perl5:/usr/local/lib/site_perl/5.8.8:/usr/local/site_p
+erl/common"
    PERL5OPT=""
    PERL5_CPANPLUS_IS_RUNNING="9961732"
    PERL5_CPAN_IS_RUNNING="9961732"
    PERL_LOCAL_LIB_ROOT="/home/perl_ss/perl5"
    PERL_MB_OPT="--install_base /home/perl_ss/perl5"
    PERL_MM_OPT="INSTALL_BASE=/home/perl_ss/perl5"
  @INC:
    /home/perl_ss/perl5/lib/perl5/aix-thread-multi
    /home/perl_ss/perl5/lib/perl5/aix-thread-multi
    /home/perl_ss/perl5/lib/perl5
    /usr/local/lib/site_perl/5.8.8/aix-thread-multi
    /usr/local/lib/site_perl/5.8.8
    /usr/local/site_perl/common
    /usr/opt/perl5/lib/site_perl/5.20.1/aix-thread-multi
    /usr/opt/perl5/lib/site_perl/5.20.1
    /usr/opt/perl5/lib/5.20.1/aix-thread-multi
    /usr/opt/perl5/lib/5.20.1
    /usr/opt/perl5/lib/site_perl/5.8.8
    /usr/opt/perl5/lib/site_perl
[download]

Comment on Unknown charnames when building Encode Select or Download Code

Replies are listed 'Best First'.
Re: Unknown charnames when building Encode by kcott (Archbishop) on Jan 19, 2017 at 13:20 UTC
G'day yulivee07, My Perl (Mac OS X): `$ perl -v \| head -2 \| tail -1 This is perl 5, version 24, subversion 0 (v5.24.0) built for darwin-th +read-multi-2level` [download] Both '`LATIN SMALL LETTER SHARP S`' and '`POUND SIGN`' are valid and work for me: $ perl -C -E 'say "\N{LATIN SMALL LETTER SHARP S}"' ß $ perl -C -E 'say "\N{POUND SIGN}"' Ł "It seems AIX perl is unable to find the \N{alpha} character. I am a bit lost here - where does perl usually search for characters like this?" Without additional code, '`alpha`' doesn't work by itself. I get the same error as you: `$ perl -C -E 'say "\N{alpha}"' Unknown charname 'alpha' at -e line 1, within string Execution of -e aborted due to compilation errors.` [download] You can use '`greek:alpha`': $ perl -C -E 'say "\N{greek:alpha}"' α I note from the source of t/Encode.t: `use charnames qw(greek);` [download] That also works for me (see charnames): $ perl -Mcharnames=greek -C -E 'say "\N{alpha}"' α "Can someone provide some debugging tips? " When running any tests, bear in mind that different versions of Perl support different levels of Unicode: `5.20.0`: perl5200delta: Unicode 6.3 now supported `5.22.0`: perl5220delta: Unicode 7.0 (with correction) is now supported `5.24.0`: perldelta: Unicode 8.0 is now supported [If you want Unicode 9.0, either get a developer release >= 5.25.3 or wait for 5.26] I note from your "`perl -V`" output, that `@INC` contains some `5.8.8` paths before `5.20.1` paths. It's possible that old versions of Unicode-related modules are being found first. Take a look in the code (`.pm`, `.t`, etc.) shown in your error messages for modules being used, then check which versions you have. For instance, I have `$ perl -E 'use charnames (); say $charnames::VERSION' 1.43` [download] For Perl 5.20.1, that should be 1.40; for 5.8.8 it should be 1.05. You can go to the latest distribution (http://search.cpan.org/~shay/perl-5.24.1/); use the dropdown list to select 5.20.1, 5.8.8, or any other version; then follow the link to the wanted module (e.g. `charnames`). You can test whether your system recognises any named characters by using their codepoints: $ perl -C -E 'say "\x{3b1} - \x{df} - \x{a3}"' α - ß - Ł Unicode::UCD might provide some useful information. For instance, to check which version of Unicode that a character first appeared in: `$ perl -MUnicode::UCD=charprops_all -E 'say charprops_all("U+$_")->{Ag +e} for qw{3b1 df a3}' V1_1 V1_1 V1_1` [download] For any of the tests above, you could try manipulating `@INC` first, e.g. move the 5.8.8. paths to the end of the list. Also, it could be helpful to know exactly what your build process is: manual, `cpan`, etc. See also: any links in http://perldoc.perl.org/perl.html with descriptions matching `/Unicode/`; http://www.unicode.org/. — Ken	[reply] [d/l] [select]
Re^2: Unknown charnames when building Encode by yulivee07 (Sexton) on Jan 19, 2017 at 15:02 UTC
Hi Ken, thank you very much for your suggestions. I updated my post with the perlcommands you suggested. I just noticed something interesting on the linux-machine wenn running perl -d t/Encode.t : `DB<8> x $uni 0 '\x{03B1}' DB<9> x is "\N{alpha}",substr($uni,0,1),"alpha does not map to symbol +'a'"; Unknown charname 'alpha' at (eval 23)[/usr/share/perl/5.18/perl5db.pl: +732] line 2, within string DB<10> x is "\N{greek:alpha}",substr($uni,0,1),"alpha does not map to +symbol 'a'"; ok 6 - alpha does not map to symbol 'a' 0 1` [download] So apparently, `\N{alpha}` is also wrong on the linux machine, but somehow the testcase does pass as "ok" anyways. Strange. Unfortunately, changing alpha to greek:alpha didn't help on the AIX machine. AIX doesn't know about greek:alpha either. I guess for now, that the Unicode-Table of linux and AIX differ. I am trying to find a way to display all availiable aliases of a unicode point. Something like this: `$unicode_module->get_char_aliases('U+03B1'); GREEK SMALL LETTER ALPHA ALPHA alpha` [download] Anyways, I will keep searching. Thanks for all your help so far.	[reply] [d/l] [select]
Re^3: Unknown charnames when building Encode by kcott (Archbishop) on Jan 20, 2017 at 15:08 UTC
Firstly, it seems I gave you something of a bum steer regarding `&Unicode::UCD::charprops_all`. I've been using it for a while and forgot that it was a fairly recent function (in terms of Perl versions). It was added in `v5.22.0`, along with some other functions, so won't be available on any of the Perl versions you're using. Sorry about that. See perl5220delta: Updated Modules and Pragmata. [Just a quick note on markup. While it's generally preferable to use '`<code>`' tags for code and data, which you've been doing, this doesn't work too well with Unicode characters (outside the ASCII range). In these cases, '`<pre>`' tags work better: for instance, you'll see '`α`' instead of '`α`'. For inline text, such as in paragraphs, '`<tt>`' tags serve the same purpose.] As I have neither Ubuntu nor AIX, I can't effectively reproduce your results. However, I looked into this a bit further and have a few other suggestions. As you successfully printed the characters from the codepoints: $ perl -C -E 'say "\x{3b1} - \x{df} - \x{a3}"' α - ß - Ł [I didn't need it, but you may need to add `-Mutf8` to get rid of the "`Wide character`" message you're seeing.] See what names you get for those characters: `$ perl -Mcharnames=:full -E 'say charnames::viacode($_) for (0x3b1, 0x +df, 0xa3)' GREEK SMALL LETTER ALPHA LATIN SMALL LETTER SHARP S POUND SIGN` [download] In terms of what you're referring to as "aliases", I suspect there's a very large number of these. Have a look at charnames; in particular, read what it says about `:full`, `:loose` and `:short`. There's a link to the algorithm for `:loose` matching, but it's horribly broken: it should be "http://www.unicode.org/reports/tr44/#Matching_Names". How `:short` is determined, is explained on the charnames page. `:full` is fairly straighforward: $ perl -C -E 'say "\N{GREEK SMALL LETTER ALPHA}"' α Based on that #Matching_Names algorithm, I then tried: `$ perl -C -E 'say "\N{greek small letter alpha}"' Unknown charname 'greek small letter alpha' at -e line 1, within strin +g Execution of -e aborted due to compilation errors.` [download] However, when I specified `-Mcharnames=:loose`, it worked: $ perl -Mcharnames=:loose -C -E 'say "\N{greek small letter alpha}"' α Bearing in mind the `:loose` algorithm, you can see there's a huge number of possibilities. Here's a few examples: $ perl -Mcharnames=:loose -C -E 'say "\N{GREEK_SMALL_LETTER_ALPHA}"' α $ perl -Mcharnames=:loose -C -E 'say "\N{GREEK-SMALL-LETTER-ALPHA}"' α $ perl -Mcharnames=:loose -C -E 'say "\N{GREEK-SMALL_LETTER-ALPHA}"' α $ perl -Mcharnames=:loose -C -E 'say "\N{greek_small_letter_alpha}"' α $ perl -Mcharnames=:loose -C -E 'say "\N{greek-small_letter-alpha}"' α $ perl -Mcharnames=:loose -C -E 'say "\N{greek small-letter alpha}"' α $ perl -Mcharnames=:loose -C -E 'say "\N{GrEeK SmAlL-LeTtEr aLpHa}"' α Now, as shown in my earlier post, I was able to use the `:short` forms directly: $ perl -C -E 'say "\N{greek:alpha}"' α $ perl -Mcharnames=greek -C -E 'say "\N{alpha}"' α They didn't work for you, but maybe these might: $ perl -Mcharnames=:short -C -E 'say "\N{greek:alpha}"' α $ perl -Mcharnames=:short,greek -C -E 'say "\N{alpha}"' α I also had a brief look at the source code for `charnames.pm` and `_charnames.pm`; although, I didn't delve into them too deeply. There's a lot of (non-POD) comments that may be of interest. Perhaps have a look at those for the versions you're using. — Ken	[reply] [d/l] [select]
Re^4: Unknown charnames when building Encode by yulivee07 (Sexton) on Jan 23, 2017 at 12:39 UTC
Re: Unknown charnames when building Encode by Anonymous Monk on Jan 19, 2017 at 10:18 UTC
Try building from zip from https://github.com/dankogai/p5-encode ( master.zip ) although I don't see anything in commits ...	[reply]
Re^2: Unknown charnames when building Encode by yulivee07 (Sexton) on Jan 19, 2017 at 12:09 UTC
Further inspection of the problem leads me to the conclusion that this may not be related to encode. I fired up my Debugger and performed some tests. Notice how the Character \N{greek:alpha} was found on Linux, but not on AIX: AIX 7.2: DB<1> use charnames qw(greek) DB<2> print "\N{alpha}" Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per +l5/lib/5.20.1/_charnames.pm line 459. [...] Unknown charname 'alpha' at (eval 9)[/usr/opt/perl5/lib/5.20.1/perl5db +.pl:732] line 2, within string DB<3> print "\N{U+03B1}" Wide character in print at (eval 10)[/usr/opt/perl5/lib/5.20.1/perl5db +.pl:732] line 2. [...] α DB<4> print "\N{greek:alpha}" Use of uninitialized value $txt in pattern match (m//) at /usr/opt/per +l5/lib/5.20.1/_charnames.pm line 459. [...] Unknown charname 'greek:alpha' at (eval 11)[/usr/opt/perl5/lib/5.20.1/ +perl5db.pl:732] line 2, within string [download] Ubuntu 14.04 `DB<1> use charnames qw(greek) DB<2> print "\N{alpha}" Unknown charname 'alpha' at (eval 8)[/usr/share/perl/5.18/perl5db.pl: +732] line 2, within string DB<3> print "\N{U+03B1}" Wide character in print at (eval 9)[/usr/share/perl/5.18/perl5db.pl:7 +32] line 2. α DB<4> print "\N{greek:alpha}" Wide character in print at (eval 10)[/usr/share/perl/5.18/perl5db.pl: +732] line 2. α` [download] Guess there is something messed up with the AIX character-table?	[reply] [d/l] [select]


Syntactic Confectionery Delight
	PerlMonks