Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Hex regex fails in subroutine

by Version7 (Novice)
on Sep 29, 2023 at 21:21 UTC ( [id://11154754]=perlquestion: print w/replies, xml ) Need Help??

Version7 has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone I've written a subroutine that replaces a handful of Windows-1252 characters within a UTF8 string with HTML entities. Basically changing ' … ' to  ' … '. This code works fine on its own outside of a function, but the regex fails within  if( $strng =~ /\xE2/ ) { it doesn't "see" \xE2. Something is changing in $clmnVal once it's passed to the subroutine and I don't understand why or how to fix it.

my $clmnVal = 'La Conner….Shelter Bay Fee Simple'; if ( grep $_ eq $clmnNm, @smblClmns ) { $clmnVal = smblRpr( $clmnVal ); } sub smblRpr { my $strng = $_[0]; my %smblCO = ( 'xE2' => { '\xE2\x80\x9A' => '‚', '\xE2\x80\x9C' => '“', '\xE2\x80\x9D' => '”', '\xE2\x80\x9E' => '„', '\xE2\x80\x93' => '–', '\xE2\x80\x94' => '—', '\xE2\x80\x98' => '‘', '\xE2\x80\x99' => '’', '\xE2\x80\xA0' => '†', '\xE2\x80\xA6' => '…', '\xE2\x84\xA2' => '™' }, 'xC2' => { '\xC2\xA9' => '©', '\xC2\xAE' => '®', '\xC2\xB1' => '±', '\xC2\xBC' => '¼', '\xC2\xBD' => '½', '\xC2\xBE' => '¾', '\xC2\xB0' => '°' } ); if( $strng =~ /\xC2/ ) { my $C2 = $smblCO{'xC2'}; while ( my($bytes, $replc) = each %$C2 ) { if( $strng =~ /$bytes/ ) { $strng =~ s/$bytes/$replc/g; } } } if( $strng =~ /\xE2/ ) { my $E2 = $smblCO{'xE2'}; while ( my($bytes, $replc) = each %$E2 ) { if( $strng =~ /$bytes/ ) { $strng =~ s/$bytes/$replc/g; } } } return $strng; } # end smblRpr

Replies are listed 'Best First'.
Re: Hex regex fails in subroutine
by choroba (Cardinal) on Sep 29, 2023 at 22:42 UTC
    What encoding do you use to save the file? When I save it as as Windows-1252 and change it to actually do something, the string gets replaced:
    La Conner….Shelter Bay Fee Simple

    Here's how the important bytes are represented:

    00000000: 2321 2f75 7372 2f62 696e 2f70 6572 6c0a #!/usr/bin/perl. 00000010: 7573 6520 7374 7269 6374 3b0a 7573 6520 use strict;.use 00000020: 7761 726e 696e 6773 3b0a 0a0a 6d79 2024 warnings;...my $ 00000030: 636c 6d6e 4e6d 203d 206d 7920 2463 6c6d clmnNm = my $clm 00000040: 6e56 616c 203d 2027 4c61 2043 6f6e 6e65 nVal = 'La Conne 00000050: 72e2 80a6 2e53 6865 6c74 6572 2042 6179 r....Shelter Bay 00000060: 2046 6565 2053 696d 706c 6527 3b0a 6d79 Fee Simple';.my
    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
      It's UTF8. The data I receive is from a rest api so CP1252 characters are UTF8 encoded. This data is going into a database that will be used for the web with UFT8 encoding so I need to replace these characters accordingly.
        If the data is always UTF-8 encoded, it might save some effort to decode that first and then look for the problem characters?

        BTW, you can do all this in a single pass, if performance matters.

        sub convert_to_html_entities { my $str= shift; utf8::decode($str); $str =~ s/[\x{201A}-\x{2122}]/ '&#'.ord($&).';' /ger; }

        You could even just wholesale replace all non-ascii characters to completely sidestep the encoding problem:

        sub convert_nonascii_to_html_entities { my $str= shift; utf8::decode($str); $str =~ s/[^\x20-\x7E]/ '&#'.ord($&).';' /ger; }
Re: Hex regex fails in subroutine
by jwkrahn (Abbot) on Sep 30, 2023 at 00:08 UTC
    my %smblCO = ( 'xE2' => { '\xE2\x80\x9A' => '‚', '\xE2\x80\x9C' => '“', '\xE2\x80\x9D' => '”', '\xE2\x80\x9E' => '„', '\xE2\x80\x93' => '–', '\xE2\x80\x94' => '—', '\xE2\x80\x98' => '‘', '\xE2\x80\x99' => '’', '\xE2\x80\xA0' => '†', '\xE2\x80\xA6' => '…', '\xE2\x84\xA2' => '™' }, 'xC2' => { '\xC2\xA9' => '©', '\xC2\xAE' => '®', '\xC2\xB1' => '±', '\xC2\xBC' => '¼', '\xC2\xBD' => '½', '\xC2\xBE' => '¾', '\xC2\xB0' => '°' } );

    You should try putting the keys in double quotes.

    my %smblCO = ( 'xE2' => { "\xE2\x80\x9A" => '‚', "\xE2\x80\x9C" => '“', "\xE2\x80\x9D" => '”', "\xE2\x80\x9E" => '„', "\xE2\x80\x93" => '–', "\xE2\x80\x94" => '—', "\xE2\x80\x98" => '‘', "\xE2\x80\x99" => '’', "\xE2\x80\xA0" => '†', "\xE2\x80\xA6" => '…', "\xE2\x84\xA2" => '™' }, 'xC2' => { "\xC2\xA9" => '©', "\xC2\xAE" => '®', "\xC2\xB1" => '±', "\xC2\xBC" => '¼', "\xC2\xBD" => '½', "\xC2\xBE" => '¾', "\xC2\xB0" => '°' } );
    Naked blocks are fun! -- Randal L. Schwartz, Perl hacker
Re: Hex regex fails in subroutine
by BillKSmith (Monsignor) on Oct 06, 2023 at 03:34 UTC
    After much study, I determined that what your program does to a file is replace non-ascii characters with their html equivalents. Your hex sequences suggest that the external file is encoded as utf8, but you read it as a string of bytes.

    It has already been shown that your approach can be made to work. I prefer the following approach because its purpose is obvious. It lets perl decode the utf8 so it can substitute characters rather than byte strings. Note that all characters are treated the same. It does not matter how many bytes they require. Note also that the hash keys are the actual characters. The code point notation is just a convenient way to specify them without needing utf8 in the Perl code.

    use strict; use warnings; use feature 'state'; use feature 'say'; #use utf8; #refer to node id=11154836 below use autodie; use Encode qw(decode); while (my $clmnVal = <>) { $clmnVal = decode('UTF-8', $clmnVal); chomp $clmnVal; $clmnVal = smblRpr( $clmnVal ); #say unpack('U0H*', $clmnVal); say $clmnVal; } exit; sub smblRpr { my $strng = $_[0]; state %smblCo = ( # code point => HTML description "\N{U+201A}" => "&#8218;", # SINGLE LOW-9 QUOTATION MARK "\N{U+201C}" => "&#8220;", # DOUBLE LEFT QUOTATION MARK "\N{U+201D}" => "&#8221;", # DOUBLE RIGHT QUOTATION MARK "\N{U+201E}" => "&#8222;", # DOUBLE LOW-9 QUOTATION MARK "\N{U+2013}" => "&#8211;", # EN DASH "\N{U+2014}" => "&#8212;", # EM DASH "\N{U+2018}" => "&#8216;", # LEFT SINGLE QUOTATION MARK "\N{U+2019}" => "&#8217;", # RIGHT SINGLE QUOTATION MARK "\N{U+2020}" => "&#8224;", # DAGGER "\N{U+2026}" => "&#8230;", # HORIZONTAL ELLIPSIS "\N{U+2122}" => "&#8482;", # TRADE MARK SIGH "\N{U+00A8}" => "&#169;", # COPYRIGHT SIGN "\N{U+00AE}" => "&#174;", # REGESTERED SIGN "\N{U+00B1}" => "&#177;", # Plus-Minus sign "\N{U+00BC}" => "&#188;", # Vulgar fraction one quarter "\N{U+00BD}" => "&#189;", # Vulgar fraction one half "\N{U+00BE}" => "&#190;", # Vulgar fraction three quarter "\N{U+00B0}" => "&#176;", # Degree sign ); state $regex = do{ my @chars = keys %smblCo; local $" = '|'; qr/@chars/; }; return ($strng =~ s/($regex)/$smblCo{$1}/gr); }

    UPDATE: Removed use utf8 from code per hippo's comment.

    Bill
      The code point notation is just a convenient way to specify them without needing utf8 in the Perl code.

      I understand this but what I don't get is why you've done this after specifying use uft8; which is the explicit declaration that you do intend to have utf8 in the Perl code. This seems contradictory to me - what am I missing here? Thanks.


      🦛

        [Nevermind. I had missed the point.]

      I would also work with the decoded strings. But I wouldn't have a table. If you have to use entities because you can't handle UTF-8, it make no sense to encode a subset only a subset of Unicode using entities. You should encode every non-ASCII Code Point.

      See HTML::Entities's encode_entities_numeric.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11154754]
Approved by 1nickt
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (4)
As of 2024-04-14 17:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found