Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:
From my reading of Advanced Perl Programming 2nd ed., best practice for utf8 handling is to mark utf8 data a such as it comes into your program and mark file handles as being utf8 when they are expected to read or write utf8.
I am using CGI.pm to process a form POST that sometimes contains utf8 data (which is urlencoded). The problem is that data retrieved by $q->param('foo') does not have the utf8 flag set when it contains utf8 data.
If I call $q->charset('utf-8') then the data is marked as utf8. But why should I have to do this?! Shouldn't the client tell the server what kind of data is being sent and CGI.pm do the right thing automatically?
Perhaps part of the problem is that I don't understand the content-type x-www-form-urlencoded. To me that content-type says nothing about what type of characterset the urlencoded data is representing.
Finally, is there any harm in calling $q->charset('utf-8') in all cases? I hate to do it but it seems necessary.
Thank you for any help.