Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

JSON encoding error

by Bod (Parson)
on May 09, 2024 at 13:31 UTC ( [id://11159356]=perlquestion: print w/replies, xml ) Need Help??

Bod has asked for the wisdom of the Perl Monks concerning the following question:

Having changed server from CentOS to Debian 12 and Perl version from 5.16.3 to 5.36.0, I am having lots of difficulties with character encoding. A topic I don't properly understand. This problem is currently manifesting itself as a failed Stripe webhook. The webhook was working fine before and now it isn't!

Stripe gives me this error:
Invalid encoding: ISO-8859-1

I have checked the Debian encoding (I think) and it says:

~# echo $LANG C.UTF-8

Here is a very cut-down version of my code to demonstrate the problem:

#!/usr/bin/perl use CGI::Carp qw(fatalsToBrowser); use strict; use warnings; use utf8; use JSON; print "Content-type: application/json; charset=UTF-8\n\n"; print encode_json { 'testing' => 'some test', };

If I call the webhook endpoint in browser, I get the expected JSON output but Stripe gives me the encoding error.

Given that only two things have changed, server OS and Perl version, it must be one of these. Is there anything I need to do on the server to ensure that it is serving UTF-8 output correctly?

There are other problems that I think are unconnected, but wise monks may find helpful as they could have the same cause...

  • .htaccess files don't work - they cause a 403 error as soon as RewriteEngine is enabled
  • When I edit some (but not all) Perl scripts, the permissions change from the Plesk psacln group and the correct owner to root
  • When duplicating or copying a Perl script the permissions always change to root and from 755 to 644
  • If I set up FTP accounts, they only have read permission regardless of what group(s) the account gets put in

Replies are listed 'Best First'.
Re: JSON UTF-8
by Corion (Patriarch) on May 09, 2024 at 13:43 UTC

    The error is likely not from the JSON but from the headers you resp. your webserver output.

    Inspect these with (for example) curl:

    curl -v https://your.server/webhook.cgi

      Thanks...I've tracked the error down - see Re^2: JSON UTF-8

      But I've learnt that I can use curl on Windows and that I can easily look at headers :)

      I don't really know what I am looking at but my untrained eye doesn't spot anything untoward in the reply from curl...

      C:\Users\User>curl -v https://test.ontheradar.uk/api/stripe/test.pl * Trying 194.164.26.100:443... * Connected to test.ontheradar.uk (194.164.26.100) port 443 * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server accepted http/1.1 * using HTTP/1.1 > GET /api/stripe/test.pl HTTP/1.1 > Host: test.ontheradar.uk > User-Agent: curl/8.4.0 > Accept: */* > * schannel: remote party requests renegotiation * schannel: renegotiating SSL/TLS connection * schannel: SSL/TLS connection renegotiated * schannel: remote party requests renegotiation * schannel: renegotiating SSL/TLS connection * schannel: SSL/TLS connection renegotiated < HTTP/1.1 200 OK < Server: nginx < Date: Thu, 09 May 2024 14:18:17 GMT < Content-Type: application/json; charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Powered-By: PleskLin < {"testing":"some test"}* Connection #0 to host test.ontheradar.uk left + intact
Re: JSON UTF-8
by hippo (Archbishop) on May 09, 2024 at 14:12 UTC
    print "Content-type: application/json; charset=UTF-8\n\n";

    FWIW, I never specify the charset= part when sending JSON because the RFC already specifies it must be UTF-8.

    Here is a very cut-down version of my code to demonstrate the problem

    Looks OK otherwise. Is this an actual data set you have tested with the integration?


    🦛

      Looks OK otherwise

      Thank you hippo for that comment...

      Your faith in the integrity of it made me go and look again at the endpoint and the settings in Stripe. I found the error which is not what Stripe was making obvious. Digging a bit deeper I found that Stripe was also saying there was 302 redirection. The webhook endpoint in Stripe was subtly wrong causing a redirect which Stripe treats as an error.

      Clearly a "senior moment" on my part

      FWIW, I never specify the charset= part when sending JSON

      Niether do I!
      That was added after I got the error from Stripe...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11159356]
Approved by choroba
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (2)
As of 2024-06-16 09:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.