|No such thing as a small change|
Re: UTF8 URI Escapingby Your Mother (Chancellor)
|on Apr 11, 2012 at 20:52 UTC||Need Help??|
I then run the usual on it ... It should be simple!
There is a big assumption out there that this stuff is so easy that you can bypass the standard libraries. This assumption only holds if you know the dozens of related RFCs inside and out and if you do, you are going to lean on someone else's implementation anyway because it will be roughly identical functionally to anything you'd write.
Eliya rightly points out that I was missing the point. So, here's a bit more answer instead of knee-jerk, use the CPAN. I am assuming the output is meant for web, though this isn’t actually stated in the OP.
Plack is necessary for this but makes it super easy to try stuff so–
Plain uri_unescape, and therefore the original code snippet, is fine if you are sending the output, bytes that are utf-8, not Perl decoded strings. The response is fine because it’s undecoded bytes.
Now with decoding to Perl’s utf-8. It doesn’t work because the output needs to be encoded to bytes and you’ll generally get errors or warnings to that effect.
Now double encoded just to see because it seems to crop up a lot when mixing approaches.
And improved/corrected versions of the CGI example. Using the -utf8 arg CGI will automatically decode things for you. This is what you want so you can deal with content correctly in regular expressions and such. It’s your responsibility to make sure the output handle is UTF-8 or that you encode to bytes. The character is right in Perl here but wrong for the output layer.
Using -CO to get utf-8 on the output layer it works fine.
Or, encoding the utf-8 to bytes.
Anyway, the first answers in the thread were, taken together, all quite thorough. This was just to have a little to play with and recant my grumpy and erroneous first stab.