Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Re: Chinese site and decoded_content() trouble

by valdez (Monsignor)
on Jun 09, 2007 at 17:54 UTC ( #620206=note: print w/replies, xml ) Need Help??

in reply to Chinese site and decoded_content() trouble

Ni hao :) It seems that the page you are requesting cannot be decoded properly; in fact if you add the raise_error parameter to decoded_content, you get an error. I used the following test program:

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use Encode qw/ is_utf8 /; my $agent = LWP::UserAgent->new(); my @tests = ( '', '', '', '', ); foreach my $uri (@tests) { eval { printf "test: %s\n", $uri; my $response = $agent->get($uri); my $dc = $response->decoded_content( raise_error => 1 ); printf "is decoded content utf8? %s\n", is_utf8($dc); }; if ($@) { print "decode failed: $@\n"; } print "\n"; }
You could try to force the charset used by adding charset parameter.

Ciao, Valerio

Replies are listed 'Best First'.
Re^2: Chinese site and decoded_content() trouble
by isync (Hermit) on Jun 11, 2007 at 08:50 UTC
    Thanks for your comment! And thank you for pointing out the raise_error switch - which I wasn't aware of (rtfm...)!

    Running your test script gives me these errors:
    test: is decoded content utf8? 1 test: decode failed: euc-cn "\x82" does not map to Unicode at /usr/lib/perl/ +5.8/ line 166. at line 22 test: decode failed: utf8 "\xE6" does not map to Unicode at /usr/lib/perl/5. +8/ line 166. at line 22 test: decode failed: euc-cn "\xA9" does not map to Unicode at /usr/lib/perl/ +5.8/ line 166. at line 22
    which points me at the problem that decoded_content uses the wrong charset for decoding. (right?)

    A search on the issue led me to this helpful thread and the non-cpan module HTTP::Response::Charset (which I haven't tried so far).

    So I am stuck here with three options:
    1. use Encode::Guess
    2. use HTML::Encoding
    3. use HTTP::Response::Charset

    A quick test seems to show that HTML::Encoding is the most reliable (and much less a hassle to install than Encode::Detect). Now I will try
    my $enco = encoding_from_http_message($resp); my $utf8 = decode($enco => $resp->content);
    But how do I combine it elegantly with LWP? (I am not a great module guru..)
    Is there a way to pull it into the LWP context and get, let's say, $response->as_utf8 (which is the result of the above)?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://620206]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (3)
As of 2016-10-24 01:20 GMT
Find Nodes?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?

    Results (302 votes). Check out past polls.