http://www.perlmonks.org?node_id=11159356

Bod has asked for the wisdom of the Perl Monks concerning the following question:

Having changed server from CentOS to Debian 12 and Perl version from 5.16.3 to 5.36.0, I am having lots of difficulties with character encoding. A topic I don't properly understand. This problem is currently manifesting itself as a failed Stripe webhook. The webhook was working fine before and now it isn't!

Stripe gives me this error:
Invalid encoding: ISO-8859-1

I have checked the Debian encoding (I think) and it says:

~# echo $LANG C.UTF-8

Here is a very cut-down version of my code to demonstrate the problem:

#!/usr/bin/perl use CGI::Carp qw(fatalsToBrowser); use strict; use warnings; use utf8; use JSON; print "Content-type: application/json; charset=UTF-8\n\n"; print encode_json { 'testing' => 'some test', };

If I call the webhook endpoint in browser, I get the expected JSON output but Stripe gives me the encoding error.

Given that only two things have changed, server OS and Perl version, it must be one of these. Is there anything I need to do on the server to ensure that it is serving UTF-8 output correctly?

There are other problems that I think are unconnected, but wise monks may find helpful as they could have the same cause...

Replies are listed 'Best First'.
Re: JSON UTF-8
by Corion (Patriarch) on May 09, 2024 at 13:43 UTC

    The error is likely not from the JSON but from the headers you resp. your webserver output.

    Inspect these with (for example) curl:

    curl -v https://your.server/webhook.cgi

      Thanks...I've tracked the error down - see Re^2: JSON UTF-8

      But I've learnt that I can use curl on Windows and that I can easily look at headers :)

      I don't really know what I am looking at but my untrained eye doesn't spot anything untoward in the reply from curl...

      C:\Users\User>curl -v https://test.ontheradar.uk/api/stripe/test.pl * Trying 194.164.26.100:443... * Connected to test.ontheradar.uk (194.164.26.100) port 443 * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server accepted http/1.1 * using HTTP/1.1 > GET /api/stripe/test.pl HTTP/1.1 > Host: test.ontheradar.uk > User-Agent: curl/8.4.0 > Accept: */* > * schannel: remote party requests renegotiation * schannel: renegotiating SSL/TLS connection * schannel: SSL/TLS connection renegotiated * schannel: remote party requests renegotiation * schannel: renegotiating SSL/TLS connection * schannel: SSL/TLS connection renegotiated < HTTP/1.1 200 OK < Server: nginx < Date: Thu, 09 May 2024 14:18:17 GMT < Content-Type: application/json; charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Powered-By: PleskLin < {"testing":"some test"}* Connection #0 to host test.ontheradar.uk left + intact
Re: JSON UTF-8
by hippo (Archbishop) on May 09, 2024 at 14:12 UTC
    print "Content-type: application/json; charset=UTF-8\n\n";

    FWIW, I never specify the charset= part when sending JSON because the RFC already specifies it must be UTF-8.

    Here is a very cut-down version of my code to demonstrate the problem

    Looks OK otherwise. Is this an actual data set you have tested with the integration?


    🦛

      Looks OK otherwise

      Thank you hippo for that comment...

      Your faith in the integrity of it made me go and look again at the endpoint and the settings in Stripe. I found the error which is not what Stripe was making obvious. Digging a bit deeper I found that Stripe was also saying there was 302 redirection. The webhook endpoint in Stripe was subtly wrong causing a redirect which Stripe treats as an error.

      Clearly a "senior moment" on my part

      FWIW, I never specify the charset= part when sending JSON

      Niether do I!
      That was added after I got the error from Stripe...