Re^2: Wierd behaviour with HTML::Entities::decode_entities()

[ikegami]

One can imagine a strange scenario where,

If the scenario you describe is possible, it's a bug in HTML::Entities. And it's not what the OP is seeing. The OP claims he need to do too many decodings, not too few.

I think you're thinking of double-encoding, where "foo" was accidentally encoded as

"foo"

when it should have been encoded as

"foo"

[JadeNB]

If the scenario you describe is possible, it's a bug in HTML::Entities. And it's not what the OP is seeing. The OP claims he need to do too many decodings, not too few.

I don't think that I'm claiming what you think I'm claiming. :-) What I meant was that, say, 
 (or even just 
) would be interpreted (incorrectly) as 
 by two passes of decode_entities, but not by one. This gives “unexpected decoding” after the second pass, but it's not a bug in HTML::Entities.

(UPDATE: I meant what I meant, but it wasn't quite what I said. A better example is a, which becomes a after one pass of the decoder and then (incorrectly) a after another. This is very particular to the ordering I mentioned earlier (first decimal, then hexadecimal, then named entities are expanded). This is the ordering in the pure-Perl decode_entities_old in HTML::Entities; I have no idea if the XS version also behaves this way. Perhaps you thought that I was mentioning that, say, &#amp;quot; " would be incorrectly converted to "? You're right, it seems to me that that is what will happen, and that it is a bug.)

On the other hand, I couldn't, and can't, think of a way that this would give the behaviour that the OP is seeing. The kind of double-encoding you mentioned sounds far more likely—and the remedy, I think, is the same, to look at the intermediate steps along the way to see where something's going wrong. (Actually, I guess that's so generic that it's true for just about any problem.)

[ikegami]

This is very particular to the ordering I mentioned earlier (first decimal, then hexadecimal, then named entities are expanded)

No, it isn't

Different nesting order:

>perl -MHTML::Entities -le"print decode_entities '"';
"

>perl -MHTML::Entities -le"print decode_entities '"';
"

Different sibling order:

[ Can't find a valid example ]

Or I still don't understand. Please given an example where ordering matters.

[JadeNB]

The source for decode_entities_old, which I thought was just a pure-Perl version of decode_entities, looks roughly like this (I've stripped out some context dependence):

sub decode_entities_old
{
    my $array = [ @_ ];
    my $c;
    for (@$array) {
    s/(&\#(\d+);?)/$2 < 256 ? chr($2) : $1/eg;
    s/(&\#[xX]([0-9a-fA-F]+);?)/$c = hex($2); $c < 256 ? chr($c) : $1/
+eg;
    s/(&(\w+);?)/$entity2char{$2} || $1/eg;
    }
    return @$array;
}

With this code, and the under-populated hash my %entity2char = ( amp => '&', quot => '"' ), we have

say decode_entities_old '"'; => "
say decode_entities_old '"'; => "

We would get (essentially) the opposite behaviour if we switched the second and third substitutions; that's what I mean by “This is very particular to the ordering”.

Of course, your example shows that decode_entities (which I guess is implemented in XS—I couldn't find it in the source) doesn't exhibit this buggy behaviour, so I guess that there's a reason that the sub I quoted has _old postpended. :-)

UPDATE: I just noticed that I'd mangled my intended bug-inducer, ", above. I wonder if decode_entities (not decode_entities_old) handles it correctly?