libwww-perl, POE and Unicode

[OverlordQ]

Seems the latest 'update' to libwww-perl broke things that used to work with POE::Component::Client::HTTP.

After the upgrade my script dies on certain requests but always in the same spot and giving the same error:
HTTP::Message content must be bytes at /home/overlordq/lib/POE/Component/Client/HTTP/Request.pm line 187
The relevant lines in PoCo::Client::HTTP are:

      eval { $content = $response->decoded_content };
      if ($content) { $response->content($content); }

Digging through HTTP::Message, this has to do with utf8 stuff in the content, and the relevant lines in there are:
In the content subroutine is a call to _set_content, and in _set_content is a call to _utf8_downgrade:

*_utf8_downgrade = defined(&utf8::downgrade) ?
    sub {
        utf8::downgrade($_[0], 1) or
            Carp::croak("HTTP::Message content must be bytes")
    }
    :
    sub {
    };

Looking through the libwww changelog I noticed: "Don't allow HTTP::Message content to be set to Unicode strings."

Any suggestions on what's a man to do? And which module has the wrong behavior so I know where to report it? FWIW, here's the headers being sent back from the server that causes the error and possibly a related cpan bug:

HTTP/1.x 200 OK
Date: Thu, 01 May 2008 02:30:39 GMT
Server: Apache
X-Powered-By: PHP/5.2.5
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Content-Language: en
Vary: Accept-Encoding,Cookie
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Encoding: gzip
Content-Length: 12241
Content-Type: text/html; charset=utf-8
Connection: keep-alive

[Juerd]

      eval { $content = $response->decoded_content };
      if ($content) { $response->content($content); }

That's bad. HTTP::Message has separate methods for this for a very good reason: An HTTP message can only ever be a sequence of octets. Together with the charset attribute of the Content-Type header, you can decode this sequence of octet, and get a Unicode string from it; the decoded_content method automates this common use. It looks like PoCoCli::HTTP is trying to be helpful and do this so that its users don't have to, but this is a bad idea because:

• It destroys the coherence between content and Content-Type;charset
• It destroys the coherence between content and Content-Length
• It destroys the original octet buffer, which might be very useful (e.g. to re-send octet-verbatim efficiently, or because the charset attribute was incorrect but you know what it should have been)
• The message is no longer actually HTTP, because HTTP as a TCP protocol is purely octet based
I guess the LWP author got tired of (hearing about) weird bugs caused by sending this kind of corrupt HTTP messages and decided to end this madness. I applaud this decision. Now PoCoCli::HTTP's author has to fix their module. Which, yes, will probably break lots of existing code using PoCoCli::HTTP that depended on this helpfulness. And again we see that innocent users are hurt by someone's ignorance of (or lack of appreciation for) the binary versus text distinction.

Remove these two lines, and use decoded_content instead of content if you need the decoded content.

[danmcb]

the new test that has been added in HTTP::Message is this:

sub {
utf8::downgrade($_[0], 1) or
Carp::croak("HTTP::Message content must be bytes")
}

Isn't suddenly deciding to die on something which previous releases didn't even test for a bit, erm, extreme? The thinking, technically speaking, may be completely correct, of course ...

[Juerd]

This is neither just a test, nor extreme. This snippet is exactly what every octet oriented Perl function should do.

Doubtless, this will "break" some existing code, as it has clearly done in this case. But believe me, clear breakage with a meaningful error message is SO much better than the vague semi-random encoding trouble you could have gotten without it!

An HTTP message is strictly a stream of octets. It is impossible to have unencoded text, because TCP does not support that. If you would try to use the rejected message over a socket, you would get Perl warnings about "wide characters", a sign that Perl is doing the best it can to force the invalid value into an octet form: it UTF-8 encodes and warns about it. However, Perl first tries to downgrade. If that succeeds, it uses the downgraded string and does not complain. This repairs several bugs caused by sloppy programming.

The snippet that you cited does the same thing: downgrade and use the downgraded form, but if it cannot be downgraded (that is: if the string contains a character with an ordinal value greater than 255) to byte form, complain. The way in which it complains is different from Perls built in. This code DIES. Your gain: you now know exactly that there is a problem, and where it is. Without this new addition to LWP you would probably not have know that there was a problem, and if you had, you would probably still be looking for the source. And if you're like most Perl programmers who don't yet fully understand the octet/text separation logic, you would probably have mis-identified something else as the source, and "fixed" it by making your code forward incompatible. Much like PoCoCli::HTTP did.

All I can say is: be glad, very glad, that LWP got this change. The pain it causes is temporarily, and much less than the pain it prevents.

[rcaputo]

POE::Component::Client::HTTP 0.84 has been released to PAUSE with a fix for this issue. The referenced rt.cpan.org ticket is closed.

Thanks everyone for an insightful discussion, but I especially appreciate OverlordQ and Yuri Karaban for discussing the issue on rt.cpan.org. By helping me keep track of the problem, they also helped me resolve it.