Cleaning up non 7-bit Ascii Chars for XML-processing

[liverpole]

Greeting fellow monks,

For the better part of this past year, I've been the principal engineer at my company on a project involving handling XML data.  I chose to use XML::Simple (with XML::LibXML::SAX for the parser), and it has served my needs quite well, even though it was my first real experience with XML processing.

Along the way, I've needed to massage input text so that XML::Simple would handle it correctly.  The XML is always output as "UTF-8", and the subroutine I was using for "cleaning" the input text looked like this:

sub xml_compliance {
    my ($self, $field) = @_;
    $field =~ s/&/&/g;
    $field =~ s/</&lt;/g;
    $field =~ s/>/&gt;/g;
    $field =~ s/"/&quot;/g;
    $field =~ s/'/&#39;/g;
    return $field;
}

Yesterday we discovered that this wasn't sufficient; an error occurred because someone gave us input containing a character which looked like an apostrophe (ascii 0x27), but was in fact an ascii 0x92 ("smart quotes", or whatever they're called), which broke the parsing.  When I say "broke the parsing", I mean that I this error from XMLin:

    Entity: line 15: parser error : Input is not proper UTF-8, indicat
+e encoding !

So I revised my subroutine as follows:

sub xml_compliance {
    my ($self, $field) = @_;

    my $fixed = "";

    my $h_translate = {
        '&' => '&',
        '<' => '&lt;',
        '>' => '&gt;',
        '"' => '&quot;',
        "'" => '&#39;',
    };

    my @chars = split //, $field;
    foreach my $char (@chars) {
        if (exists $h_translate->{$char}) {
            $fixed .= $h_translate->{$char};
        } elsif (ord($char) > 0x7f) {
            # This is where we handle all non 7-bit ascii
            $fixed .= sprintf "&#%02x;", ord($char);
        } else {
            $fixed .= $char;
        }
    }

    return $fixed;
}

That is, it will handle those "smart quotes" by converting them to "\", as well as any other ascii characters with the high bit turned on.

My questions are:

1. Is this an appropriate way to handle such characters?
2. Is there some simpler/easier/better way to do it?
3. Is there some module for this which I'm unaware of?
4. Is there anything else I should be taking into account?

Thanks in advance for any enlightenment!

[ikegami]

You're getting stuff as cp1252 — "’" is 92 in cp1252 — but you're outputting it as is in a document you claim is UTF-8.

Always decode your inputs. Always encode your outputs. You are apparently doing neither.

Note that the quote is character U+2019, so the proper escape is ’ or ’, not \.

If you pass properly decoded text to the following function, it will produce 7-bit clean UTF-8 (aka US-ASCII) XML text and XML attribute values.

sub encode_entities {
    my ($self, $text) = @_;
    $text =~ s/&/&/g;
    $text =~ s/</&lt;/g;
    $text =~ s/>/&gt;/g;
    $text =~ s/"/&quot;/g;
    $text =~ s/'/&#39;/g;
    $text =~ s/([^\x20-\x7E])/sprintf("&#x%X;", ord($1))/eg;
    return $text;
}

[liverpole]

Okay, thanks for the information.

But this line:

    $field =~ s/([^\x20-\x7E])/sprintf("&#x%X;", ord($1))/eg;

Won't that convert anything greater than or equal to a "space" (ascii 0x20) up to 0xfe?  And why are you skipping 0xff?

And other than those things, isn't that really equivalent to what I was doing?  Except that, it appears you are outputting ’ where I was outputting \, and are those in fact equivalent?

Update:  I just realized I missed the fact that your regex has '^', so it negates those characters.  That makes a lot more sense.

But I would still ask, isn't the functionality of this subroutine the same as my original (other than the extra 'x' in your output "#xNN;", which I still think is wrong), or am I missing something else?

[ikegami]

00-1F and 7F aren't printable, so I escape those too.

And other than those things, isn't that really equivalent to what I was doing?

Yes, it's the same except where it's not.

Except that, it appears you are outputting ’ where I was outputting \, and are those in fact equivalent?

So you still haven't fixed your bug. Start with that.

and are those in fact equivalent?

No. I guess that's yet another bug.

• You're not decoding your inputs. (Not yet fixed.)
• You're not encoding your outputs. (Not yet fixed.)
• You confuse 92 hex for 92 decimal. (Fixed by using the function I posted.)
• You're not outputting 7-bit clean as desired. (Fixed by using the function I posted.)

[grantm]

You might want to pass your data through Encoding::FixLatin to ensure it's converted from the CP1252 data you seem to have, into UTF8.

I'm not clear at all on where in the process your xml_compliance routine is being used, nor why you need it.

[aquarium]

for me this would always be a matter of defining the acceptable characters...and if any part of input fails to meet criteria, then reject entire input. a human needs to check and fix the file/node. trying to instead cover all the cases of what's not acceptable, can be quite elusive, and prone to break in future.