Re^3: parsing XML fragments (xml log files) with... a regex

[ikegami]

For those interested, it can't handle

• Numerical entities (decimal and hex).*
• External entities (e.g. HTML's &eacute).*
• Character decoding.**
• UTF-16, UTF-32, UCS-2, UCS-4.**
• CDATA.
• Namespace prefixes. (They're included as part of the name.)***
• Comments.
• Identification of an element's namespace.***
• XML validation (i.e. it allows some malformed XML).
• (more? this wasn't a thorough analysis)

Up to you to decide if it fits your needs or not.

* — A post-processor could fix this if no entities were processed at all.

** — A pre-processor such as the following would fix this:

sub _predecode {
    my $enc;
    if    ( $_[0] =~ /^\xEF\xBB\xBF/ ) { $enc = 'UTF-8';    }
    elsif ( $_[0] =~ /^\xFF\xFE/     ) { $enc = 'UTF-16le'; }
    elsif ( $_[0] =~ /^\xFE\xFF/     ) { $enc = 'UTF-16be'; }
    elsif (substr($_[0], 0, 100) =~ /^[^>]* encoding="([^"]+)"/) { $en
+c = $1; }
    else                               { $enc = 'UTF-8';    }
    return decode($enc, $_[0], Encode::FB_CROAK | Encode::LEAVE_SRC);
}

*** — A post-processor could fix this, but one wasn't supplied.

Update: Added pre-processor I had previously coded.

[tye]

It parses numerical entities. Decoding them wasn't required and would only require one regex be added. It handles namespace prefixes exactly as I wanted it to (they're included in the tag name). It is trivial to make it handle them differently (which is the point). I won't go into "validation" here, it being a subject worthy of a lengthy write-up.

A pre/post processor could fix this...

Wow. You are really stuck in thinking in terms of an XML-parsing module. There is no need to do anything in a pre-/post-processor -- which is part of the whole point of the exercise.

For example, supporting comments is 2 minutes' work and easily fits into the existing structure. The few items that rise to the level of being interesting to implement are the things that I've never actually seen used in any XML. So it shouldn't be surprising that I didn't bother to implement them in the code that implemented just what I needed for one project.

[ikegami]

It handles namespace prefixes exactly as I wanted it to

So? All I did was identify the features others might have to add to suit their needs.

It would be totally useless to me, for example, since the prefix isn't uniform across the documents I deal with. In fact, I've never encountered a situation where it was better to keep the prefixes.

For example, supporting comments is 2 minutes' work

I'm well aware that the changes are easy.

There is no need to do anything in a pre-/post-processor

Duh. There are many ways of changing it. I just suggested one.

So it shouldn't be surprising that I didn't bother to implement them in the code that implemented just what I needed for one project.

It's not. Why are you so defensive when I tell other people what they might need to adjust to suit their needs?

[tye]

Why are you so defensive when I reply to your reply? I point out things that you didn't point out and that your response actually points against and you feel the need to reply with "Duh! I knew that" and to call me defensive.

But you also didn't comment on the main point of my reply: that you seem to have missed the main point of my original posting. Doing pre-/post-processing is mostly counter to the whole point of what I was providing a demonstration of. Restricting yourself to solutions that don't touch the code is perfectly opposed to the purpose of code that was offered up as "here is code you can easily change". It isn't just "one way to fix it", it is closer to a misuse of the example.

So? All I did was identify the features others might have to add to suit their needs.

Yes, and that in itself is actually going against the point of what I wrote. I wasn't providing a subroutine for people to just use. I was providing an example of code that somebody could similarly create to meet their own needs or that somebody might choose to customize to meet their needs.

If the prospective consumer of this example code is unable to determine on their own what the code handles, then they should not be trying to use the code.

You replied exactly like one would reply to "Here is a black box that claims to parse XML but only partially". Treating it like a black box is counter to the whole purpose and hints that I failed at conveying the point to you and thus likely to others and so some clarifications were in order.

Pointing out how the code can be easily modified was not being defensive; it was trying to get back to / reinforce the original point.