Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Cleaning up non 7-bit Ascii Chars for XML-processing

by liverpole (Monsignor)
on Nov 11, 2010 at 18:47 UTC ( #870912=perlquestion: print w/ replies, xml ) Need Help??
liverpole has asked for the wisdom of the Perl Monks concerning the following question:

Greeting fellow monks,

For the better part of this past year, I've been the principal engineer at my company on a project involving handling XML data.  I chose to use XML::Simple (with XML::LibXML::SAX for the parser), and it has served my needs quite well, even though it was my first real experience with XML processing.

Along the way, I've needed to massage input text so that XML::Simple would handle it correctly.  The XML is always output as "UTF-8", and the subroutine I was using for "cleaning" the input text looked like this:

sub xml_compliance { my ($self, $field) = @_; $field =~ s/&/&amp;/g; $field =~ s/</&lt;/g; $field =~ s/>/&gt;/g; $field =~ s/"/&quot;/g; $field =~ s/'/&#39;/g; return $field; }

Yesterday we discovered that this wasn't sufficient; an error occurred because someone gave us input containing a character which looked like an apostrophe (ascii 0x27), but was in fact an ascii 0x92 ("smart quotes", or whatever they're called), which broke the parsing.  When I say "broke the parsing", I mean that I this error from XMLin:

Entity: line 15: parser error : Input is not proper UTF-8, indicat +e encoding !

So I revised my subroutine as follows:

sub xml_compliance { my ($self, $field) = @_; my $fixed = ""; my $h_translate = { '&' => '&amp;', '<' => '&lt;', '>' => '&gt;', '"' => '&quot;', "'" => '&#39;', }; my @chars = split //, $field; foreach my $char (@chars) { if (exists $h_translate->{$char}) { $fixed .= $h_translate->{$char}; } elsif (ord($char) > 0x7f) { # This is where we handle all non 7-bit ascii $fixed .= sprintf "&#%02x;", ord($char); } else { $fixed .= $char; } } return $fixed; }

That is, it will handle those "smart quotes" by converting them to "&#92;", as well as any other ascii characters with the high bit turned on.

My questions are:

  1. Is this an appropriate way to handle such characters?
  2. Is there some simpler/easier/better way to do it?
  3. Is there some module for this which I'm unaware of?
  4. Is there anything else I should be taking into account?

Thanks in advance for any enlightenment!


s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/

Comment on Cleaning up non 7-bit Ascii Chars for XML-processing
Select or Download Code
Re: Cleaning up non 7-bit Ascii Chars for XML-processing
by ikegami (Pope) on Nov 11, 2010 at 19:06 UTC

    You're getting stuff as cp1252 — "’" is 92 in cp1252 — but you're outputting it as is in a document you claim is UTF-8.

    Always decode your inputs. Always encode your outputs. You are apparently doing neither.

    Note that the quote is character U+2019, so the proper escape is &#x2019; or &#8217;, not &#92;.

    If you pass properly decoded text to the following function, it will produce 7-bit clean UTF-8 (aka US-ASCII) XML text and XML attribute values.

    sub encode_entities { my ($self, $text) = @_; $text =~ s/&/&amp;/g; $text =~ s/</&lt;/g; $text =~ s/>/&gt;/g; $text =~ s/"/&quot;/g; $text =~ s/'/&#39;/g; $text =~ s/([^\x20-\x7E])/sprintf("&#x%X;", ord($1))/eg; return $text; }
      Okay, thanks for the information.

      But this line:

      $field =~ s/([^\x20-\x7E])/sprintf("&#x%X;", ord($1))/eg;

      Won't that convert anything greater than or equal to a "space" (ascii 0x20) up to 0xfe?  And why are you skipping 0xff?

      And other than those things, isn't that really equivalent to what I was doing?  Except that, it appears you are outputting &#x92; where I was outputting &#92;, and are those in fact equivalent?

      Update:  I just realized I missed the fact that your regex has '^', so it negates those characters.  That makes a lot more sense.

      But I would still ask, isn't the functionality of this subroutine the same as my original (other than the extra 'x' in your output "#xNN;", which I still think is wrong), or am I missing something else?


      s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/
        00-1F and 7F aren't printable, so I escape those too.

        And other than those things, isn't that really equivalent to what I was doing?

        Yes, it's the same except where it's not.

        Except that, it appears you are outputting ’ where I was outputting \, and are those in fact equivalent?

        So you still haven't fixed your bug. Start with that.

        and are those in fact equivalent?

        No. I guess that's yet another bug.

        • You're not decoding your inputs. (Not yet fixed.)
        • You're not encoding your outputs. (Not yet fixed.)
        • You confuse 92 hex for 92 decimal. (Fixed by using the function I posted.)
        • You're not outputting 7-bit clean as desired. (Fixed by using the function I posted.)
Re: Cleaning up non 7-bit Ascii Chars for XML-processing
by grantm (Parson) on Nov 11, 2010 at 20:15 UTC

    You might want to pass your data through Encoding::FixLatin to ensure it's converted from the CP1252 data you seem to have, into UTF8.

    I'm not clear at all on where in the process your xml_compliance routine is being used, nor why you need it.

Re: Cleaning up non 7-bit Ascii Chars for XML-processing
by aquarium (Curate) on Nov 12, 2010 at 04:27 UTC
    for me this would always be a matter of defining the acceptable characters...and if any part of input fails to meet criteria, then reject entire input. a human needs to check and fix the file/node. trying to instead cover all the cases of what's not acceptable, can be quite elusive, and prone to break in future.
    the hardest line to type correctly is: stty erase ^H

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://870912]
Approved by ikegami
Front-paged by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (13)
As of 2014-10-31 15:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (219 votes), past polls