http://www.perlmonks.org?node_id=877382

mcm1303 has asked for the wisdom of the Perl Monks concerning the following question:

I wrote an ajax service in perl which needs to be able to send a JSON message to a variety of browsers.

I have found that if JSON messages contain anything other than 7-bit ascii, i.e. no special characters, IE8 will fail to eval(). A particular character that IE chokes on is é or egrav in HTML. I had originally written the service in ASP. The tool I used to generate the JSON from ASP would interpret é into \u00E9. It seems like that approach made both Firefox and IE happy. Is there an existing perl module that will convert special characters into the \uXXXX form?

Thanks! Marlin

Replies are listed 'Best First'.
Re: Handling special characters in JSON
by sundialsvc4 (Abbot) on Dec 15, 2010 at 21:44 UTC

    A Google search of perl json special characters produced a wealth of suggestions.

    (I do not mean any “slight” by making this particular suggestion.   This is not meant as a “JGI” or “RTFG” response.)

    Several of the articles discuss your options on both the client and the server side.   Frankly, I think that the best long-term solution to your problem will include choosing a good, robust client-side JavaScript tool that will take-on the responsibility of handling recalcitrant software ... particularly “IE-anything.”   (And, let’s face it, the bugaboo is always going to be Internet Explorer, in some incarnation or another...   You need to attack the problem from both the client and the server side of the fence, and to shove the client-side burden onto a third party JavaScript-kit provider.   (Don’t worry... they’re used to it.)

      Thanks sundials. I've become accustomed to using plain old eval() but I'll try your suggestion. In addition to hopefully solving my problem client-side solutions that go beyond eval() will no doubt be more secure because malicious code can be embedded in JSON.

        Yes, “simple eval()” is Not A Good Thing ... and “Doing It Yourself” is Definitely Not.

        I have read plenty of documents, for ExtJS and Prototype and several other JavaScript platforms, which describe all the whys and wherefores of what they actually do (and, over time, continue to have to do...), just to make it work at all.   And I come away saying, “Good Grief!   Better You Than Me!”   When I read stuff that says, “Internet Explorer 7 Service-Pack 2 does it this way, but Service-Pack 3 does it that way” ... why, it made me want to switch religions.   ;-)

        It really is a testament to how good these folks are, that not only does it work “at all,” but it actually works well.   Just be sure to do exactly what they tell you to do.   These are well-traveled waters... full of now-familiar rocks.   “Where the well-worn chart tells you to turn left ... turn left, (pray,) and don’t ask questions.”   :-D

        Some JS frameworks are very lean-and-mean.   Others are, “it slices, it dices, it even makes Julienne fries... on sale today!”   But I guess that comes with the territory.   (And who knows, maybe you want to make Julienne fries!)   There are also Perl frameworks and JS frameworks that have been crafted to work well together, with fairly high-level support on the Perl side for doing complicated things on the JS side.   Lotsa things to think about, mebbe.

Re: Handling special characters in JSON
by kennethk (Abbot) on Dec 15, 2010 at 22:11 UTC
    When I've dealt with this sort of issue, I always escaped my strings before feeding them into JSON. Usually this means simply feeding it through HTML::Entities' encode_entities (or possibly URI::Escape's uri_escape_utf8 depending on its end point). I trust browsers' and JavaScript engines' non-ascii character handling about as far as I can throw them, which isn't far considering their non-corporeal nature.
Re: Handling special characters in JSON (TFM)
by tye (Sage) on Dec 16, 2010 at 02:17 UTC

    Well, if nobody else will RTFM, I will: JSON::XS. :)

    perl -MJSON::XS -del # vvvvvvvv DB<1> x JSON::XS->new()->ascii(1)->encode(["\x{00e9}"]) 0 '["\\u00e9"]' # ^^^^^^^^

    - tye        

      That assumes a perl data structure as the source, which wasn't clearly (to me) the case. Anyway, I do:
      JSON::to_json( ["\x{00e9}"], { 'ascii' => 1 } )
      (with JSON using JSON::XS).
      --
      A math joke: r = | |csc(θ)|+|sec(θ)|-||csc(θ)|-|sec(θ)|| |
      Online Fortune Cookie Search
      Office Space merchandise
        That assumes a perl data structure as the source, which wasn't clearly (to me) the case.

        Oh. Well, if only there was a way to take JSON and turn it into a Perl data structure... q-:

        But "I wrote an ajax service in perl" sure sounds to me like something that is likely to be generating JSON from Perl data structures. :)

        - tye        

Re: Handling special characters in JSON
by Anonyrnous Monk (Hermit) on Dec 15, 2010 at 23:00 UTC
    Is there an existing perl module that will convert special characters into the \uXXXX form?

    You could do something like this:

    my $s = "téxt"; $s =~ s/([^\x20-\x7e])/sprintf '\u%04x', ord($1)/ge; print $s; # t\u00e9xt

    This will convert everything outside of the printable ASCII range to \uXXXX notation.

    You might need to decode your input strings into Perl's internal Unicode representation, depending on which encoding they're originally in.