Re: Fixing broken character encoding

[Anonymous Monk]

It is possible if its been malformed once (single step), multiple iterations can be impossible.

IIRC I think http://validator.w3.org/ can help ( Bundle::W3C::Validator )

As can these
HTML::Encoding - Determine the encoding of HTML/XML/XHTML documents
Encode::Detective - detect a data encoding
Encoding::FixLatin - takes mixed encoding input and produces UTF-8 output
Encode::DoubleEncodedUTF8 - Fix double encoded UTF-8 bytes to the correct one

But you ought to post some minimal html

I figure it ought to be as simple as parsing the file, decoding the entities, treating the string as octets and deciding what charset it is

[Anonymous Monk]

This could work

#!/usr/bin/perl --
use strict; use warnings;
use Data::Dump        qw' dd                              ';
use HTML::Entities    qw' encode_entities decode_entities ';
use Encode            qw' encode decode                   ';
use Encode::Detective qw' detect                          ';

my $odata = my $str = '’';

decode_entities($str);
dd $str;
dd encode_entities($str);

my $encoding = detect($str);
dd $encoding;

$str = decode( 'UTF-8', $str );
dd $str;
dd encode_entities($str);

__END__
"\xE2\x80\x99"
"’"
"UTF-8"
"\x{2019}"
"’"

[pfaut]

That's showing some promise. Thank you.