Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^2: Decoding, Encoding string, how to?

by way (Sexton)
on Apr 03, 2009 at 05:51 UTC ( #755179=note: print w/ replies, xml ) Need Help??


in reply to Re: Decoding, Encoding string, how to? (internal encoding)
in thread Decoding, Encoding string, how to?

Your graphic is really helpful to understand how the encode works on Perl

you has been really clear, just one more thing, if a want to print using iso-8859-1 it could be possible downgrading, because it changes the internal encoding to this last one and when I print the string (in normal case), i'll have an iso-8859-1 text in the output, isn't it?

I was checked, the examples, and I performed other test using the graphic like downgrading, but i couldn't print the original (U+201C, LEFT DOUBLE QUOTATION MARK), thinking, I see that it's not representing in the iso-8859-1 charset, but I found different issues regarding that:

1- If I downgrade the string, perl dies with a message that has wide characters and I guest, that's important becouse in other case it could be cut the internal string without notice, in fact, we can check if it's downgradeable or not using:

my $str = "This's a \x{201c}test\x{201d}"; unless (utf8::downgrade($str, 1)) { die "Isn't downgradable\n"; }

2- using :encoding on an output stream i can see two notice in this case, about perl can't map to iso-8859-1 but in the output appear the unmapped character as an string like \x{201c}.

3- using Encode::encode the unmapped character is printed as an ? question symbol and not report any notice

Thank you so much, is a great explanation


Comment on Re^2: Decoding, Encoding string, how to?
Select or Download Code
Re^3: Decoding, Encoding string, how to?
by ikegami (Pope) on Apr 03, 2009 at 08:50 UTC
    I see question marks, but I'm not sure if there's a question in there. You seem to have a good grasp of the concept.

    if a want to print using iso-8859-1 it could be possible downgrading

    You'd get the right result, at the cost of confusing your readers. You'd be saying you're doing one thing (changing the internal format) while actually doing another (changing the encoding of the string).

    I see that it's not representing in the iso-8859-1 charset

    Correct, iso-8859-1 cannot encode U+201C. cp1252 can. cp1252 is Microsoft's extension of iso-8859-1. It's a commonly used encoding in the Windows world, which is why U+201C is encountered frequently.

    we can check if it's downgradable or not using utf8::downgrade($str, 1)

    Indeed. I have used that very code to make sure a sub was only given bytes before calling a function that expects to only get bytes. At the same time, it makes sure the bytes aren't internally encoded as UTF-8. Most XS functions can't handle that (which is really a bug in the XS function).

    using Encode::encode the unmapped character is printed as an ? question symbol and not report any notice

    How encode handles errors is configurable using its third parameter.

      You'd get the right result, at the cost of confusing your readers. You'd be saying you're doing one thing (changing the internal format) while actually doing another (changing the encoding of the string).

      Yes, my example is for understand the basic step of how perl works internally and what I can obtain handle it, in the mayor part it is a theory test, because it must be handle usually with functions like Encode::encode

      Correct, iso-8859-1 cannot encode U+201C. cp1252 can. cp1252 is Microsoft's extension of iso-8859-1, and it's commonly used encoding in the Windows world. That's why U+201C is encountered frequently.

      You right, i really don't knew it but checking the cp1252 i can see the character 201C.

      Here you have http://en.wikipedia.org/wiki/Windows-1252

      Indeed. I have used that very code to make sure a sub was only given bytes before calling a function that expects to only get bytes. At the same time, it makes sure the bytes aren't internally encoded as UTF-8. Most XS functions can't handle that (which is really a bug in the XS function).

      That's really important, i'll take mental note, of common error using XS functions.

      You're very helpful to undestand this topic, it's small but clear thank you again

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://755179]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2014-11-28 04:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (193 votes), past polls