Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re^6: Encoding/decoding question

by slugger415 (Scribe)
on Sep 12, 2011 at 20:30 UTC ( #925566=note: print w/replies, xml ) Need Help??

in reply to Re^5: Encoding/decoding question
in thread Encoding/decoding question

If I set encoding to ASCII, it converts it to this:

is that correct for the accented e? It still comes out wonky after my Perl script gets ahold of it:


Replies are listed 'Best First'.
Re^7: Encoding/decoding question
by tchrist (Pilgrim) on Sep 12, 2011 at 21:23 UTC
    The input is UTF-8, but you are treating it as Latin-1. You cant do that. That is why you are getting that sort of output.
      Sorry to be dumb here, but where am I treating it as Latin-1? How do I change it? -- Scott
        I am processing some XHTML pages (using XML::Twig) that contain numerous character entities, such as:
        Sorry to be dumb here, but where am I treating it as Latin-1? How do I change it?
        The problem is in your original XHTML file, because it has the literal 12‑byte ASCII sequence
        1. &
        2. #
        3. 1
        4. 9
        5. 5
        6. ;
        7. &
        8. #
        9. 1
        10. 6
        11. 9
        12. ;
        So something somewhere somewhen took a UTF‑8 file and replaced not each complete multibyte character with its single entity, but rather each individual component byte as the Latin‑1 code point number.

        This may have happened because some program read an undecoded binary byte stream and never decoded it before trying to convert non‑ASCII into numeric entities. For example, here I use -CS in the first process to say its UTF‑8 but then lie to the second one by using -C0 to say that it isnt. That would produce the sort of thing that you saw:

        $ perl -CS -le 'print "na\x{EF}ivet\x{E9}"' navet $ perl -CS -le 'print "na\x{EF}ivet\x{E9}"' | perl -C0 -pe 's/(\P{ASCII})/"&#".ord($1).";"/ge' naïiveté $ perl -CS -le 'print "na\x{EF}ivet\x{E9}"' | perl -C0 -pe 's/(\P{ASCII})/sprintf "&#x%02X;", ord($1)/ge' naïiveté
        Compare with the right answers:
        $ perl -CS -le 'print "na\x{EF}vet\x{E9}"' | perl -CS -pe 's/(\P{ASCII})/"&#".ord($1).";"/ge' naïveté $ perl -CS -le 'print "na\x{EF}vet\x{E9}"' | perl -CS -pe 's/(\P{ASCII})/sprintf "&#x%02X;", ord($1)/ge' naïveté
        So what you really need to do is track down whatever errant procedure caused this mess in the first place, and fix that, since it will never work right that way.

        This demonstrates putting it back to UTF-8:

        $ perl -CS -le 'print "na\x{EF}vet\x{E9}"' | perl -C0 -pe 's/(\P{ASCII})/"&#".ord($1).";"/ge' | perl -C0 -pe 's/&#(\d+);/chr($1)/ge' navet
        And this, heaven help you, demonstrates doing that and then doing the entities the right way around in the first place:
        $ perl -CS -le 'print "na\x{EF}vet\x{E9}"' | perl -C0 -pe 's/(\P{ASCII})/"&#".ord($1).";"/ge' | perl -MEncode -C0 -pe 's/&#(\d+);/chr($1)/ge; $_ = decode_utf8($_, 1); s/(\P{ASCII})/"&#".ord($1).";"/ge' naïveté
        That means that if you were courageous enough, you could just do this:
        $ perl$$ -MEncode -C0 -pe 's/&#(\d+);/chr($1)/ge; $_ += decode_utf8($_, 1); s/(\P{ASCII})/"&#".ord($1).";"/ge' all*your*bro +ken*files.xhtml
        Heres a version that runs from as a script instead of from the command line:
        #!/usr/bin/env perl use strict; use warnings; use Encode; die "gimme args" unless @ARGV; $^I = "$$"; while (<>) { s/&#(\d+);/chr($1)/ge; $_ = decode_utf8($_, 1); s/(\P{ASCII})/ "&#" . ord($1) . ";" /ge; print; }
        No warantees, though. Make sure you thoroughly understand all this before you further mangle your files.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://925566]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (3)
As of 2018-01-21 07:08 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (227 votes). Check out past polls.