XML::RSS

[alexg]

Hi, I'm writing a quick and dirty RSS feed aggregator and I'm getting very frustrated with one tiny problem. Occasionally the RSS XML docs contain characters which cause XML::Parser to choke.

The XML::Parser error message generated is:
"not well-formed (invalid token) at line 15, column 19, byte 530"

when I look at byte 530 it turns out to be the 'é' in Nescafé. Other exotic characters also cause XML::Parser to stop dead. I've tried the nice_string function from the Unicode man page:

sub nice_string {
join("",
        map { $_ > 255 ? # if wide character...
        sprintf("\\x{%04X}", $_) :  # \x{...}
        chr($_) =~ /[[:cntrl:]]/ ?  # if control character
        sprintf("\\x%02X", $_) :    # \x..
    chr($_)                     # else as themselves
    } unpack("U*", $_[0]));     # unpack Unicode
}

but this enrages XML::Parser even further and it fails and the first end-of-line character. I'm using LWP::Simple to grab the XML so my script essentially looks like this:

my $rss = new XML::RSS;
eval { $rss->parse(nice_string(get($url))); };

Can anyone recomend a module/function that will reliably sanitise the string that get() returns, in an encoding suitable for XML::Parser?

PS I've looked at the 'encoding' option for XML::Parser but it doesn't seem to change the result :(

[merlyn]

Sounds to me like the RSS is incorrectly claiming that it is UTF-8 when in fact it is Latin-1. That's a common problem when people are writing RSS with hand-rolled ad-hoc tools instead of formal DOM tools.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

[alexg]

Ok, I've done some more reading - let me see if I've got this right:

LWP::Simple returns HTML which is encoded in ISO-8859-1 (Latin-1).
XML::Parser defaults to UTF-8 (unicode) if the XML does not specify an encoding.

So using:
my $rss = new XML::RSS( 'encoding' => 'ISO-8859-1' );

should work.

It doesn't.

[rjray]

Specifying the encoding like that is only used when creating an RSS stream using the library. It won't affect a parsed input stream's encoding.

--rjray

[zby]

Have you tried to just set the correct encoding in the XML? Something like this:  <?xml encoding='ISO-8859-1'>.

[alexg]

Yeah the XML docs all claim to be ISO-8859-1, even though they don't seem to be...

[zby]

E acute is in ISO-8859-1, perhaps it is not right encoded? From the XML::Parser documentation: "The built-in encodings are: UTF-8, ISO-8859-1, UTF-16, and US-ASCII." so this is not the case that ISO-8859-1 is not handled by the parser.

By the way when you make it UTF-8 and it claims to be ISO-8859-1 you don't fix anything.

[AnthonyLewis]

I'm currently working on a content management system that allows users to enter text that is stored in XML files. Everything worked great until a Spanish speaking user tried it.

I stole this snippet from the Perl-XML FAQ at http://perl-xml.sourceforge.net/faq/

$str =~ s/([^\x0A-\x7F])/'&#' . ord($1) . ';'/gse;

Run your data through this before sending it to the parser and things might work better...

[ajt]

alexg,

Malformed XML is the bane of RSS. According to Mark Pilgrim about 10% of typical RSS feeds are malformed*, indeed the UK IT publication The Register has usable XML for only a few days in a given month.

You will find a wide range of problems that will cause XML::Parser the core of XML::RSS to explode:

• Data encoded in one format, but declared in another (or in default utf-8).
• Junk before the start XML declaration, the CMS Vignette tends to do this, and it's popular with big companies.
• Badly nested tags, the CMS is sloppy at non-well formness checking, so it comes out and goes into the RSS feed broken.
• Inproperly escaped ampersands and entities are a very common problem too.

In this node "How do I clean RSS feeds to make them usable?", Matts suggested his rssmirror, the guts of which are now included in both XML::RSS and XML::RSS::Tools.

I became so annoyed with bad XML in RSS feeds that I wrote XML::RSS::Tools to deal with the problems I found, which led to brian d foy taking over XML::RSS fixing a lot of it's problems, and with time designing a whole new version.

See also:

• Parsing badly formed RSS or XML

Good Luck!

* Parsing RSS At All Costs


--
ajt