in reply to Re^2: Wierd behaviour with HTML::Entities::decode_entities()
in thread Wierd behaviour with HTML::Entities::decode_entities()

If the scenario you describe is possible, it's a bug in HTML::Entities. And it's not what the OP is seeing. The OP claims he need to do too many decodings, not too few.

I don't think that I'm claiming what you think I'm claiming. :-) What I meant was that, say, 
 (or even just 
) would be interpreted (incorrectly) as 
 by two passes of decode_entities, but not by one. This gives “unexpected decoding” after the second pass, but it's not a bug in HTML::Entities.

(UPDATE: I meant what I meant, but it wasn't quite what I said. A better example is a, which becomes a after one pass of the decoder and then (incorrectly) a after another. This is very particular to the ordering I mentioned earlier (first decimal, then hexadecimal, then named entities are expanded). This is the ordering in the pure-Perl decode_entities_old in HTML::Entities; I have no idea if the XS version also behaves this way. Perhaps you thought that I was mentioning that, say, &#amp;quot; " would be incorrectly converted to "? You're right, it seems to me that that is what will happen, and that it is a bug.)

On the other hand, I couldn't, and can't, think of a way that this would give the behaviour that the OP is seeing. The kind of double-encoding you mentioned sounds far more likely—and the remedy, I think, is the same, to look at the intermediate steps along the way to see where something's going wrong. (Actually, I guess that's so generic that it's true for just about any problem.)

Replies are listed 'Best First'.
Re^4: Wierd behaviour with HTML::Entities::decode_entities()
by ikegami (Pope) on Dec 14, 2009 at 16:27 UTC

    This is very particular to the ordering I mentioned earlier (first decimal, then hexadecimal, then named entities are expanded)

    No, it isn't

    Different nesting order:

    >perl -MHTML::Entities -le"print decode_entities '"'; " >perl -MHTML::Entities -le"print decode_entities '"'; "

    Different sibling order:

    [ Can't find a valid example ]

    Or I still don't understand. Please given an example where ordering matters.

      The source for decode_entities_old, which I thought was just a pure-Perl version of decode_entities, looks roughly like this (I've stripped out some context dependence):

      sub decode_entities_old { my $array = [ @_ ]; my $c; for (@$array) { s/(&\#(\d+);?)/$2 < 256 ? chr($2) : $1/eg; s/(&\#[xX]([0-9a-fA-F]+);?)/$c = hex($2); $c < 256 ? chr($c) : $1/ +eg; s/(&(\w+);?)/$entity2char{$2} || $1/eg; } return @$array; }
      With this code, and the under-populated hash my %entity2char = ( amp => '&', quot => '"' ), we have
      say decode_entities_old '&#x26;amp;quot;'; => &quot; say decode_entities_old '&amp;#x26;quot;'; => &#x26;quot;
      We would get (essentially) the opposite behaviour if we switched the second and third substitutions; that's what I mean by “This is very particular to the ordering”.

      Of course, your example shows that decode_entities (which I guess is implemented in XS—I couldn't find it in the source) doesn't exhibit this buggy behaviour, so I guess that there's a reason that the sub I quoted has _old postpended. :-)

      UPDATE: I just noticed that I'd mangled my intended bug-inducer, &&#x71;uot;, above. I wonder if decode_entities (not decode_entities_old) handles it correctly?

        If HTML::Entities were to decode &#x26;amp;quot; to &quot;, it would be buggy. I did understand you correctly. Does what I said make more sense now?

        I wonder if decode_entities (not decode_entities_old) handles it correctly?

        I don't know if that's legal in SGML/HTML — unescaped ampersand — but yes.

        $ perl -MHTML::Entities -le'print decode_entities "&&#x71;uot;"' &quot;