JSON encoding error

[Bod]

Having changed server from CentOS to Debian 12 and Perl version from 5.16.3 to 5.36.0, I am having lots of difficulties with character encoding. A topic I don't properly understand. This problem is currently manifesting itself as a failed Stripe webhook. The webhook was working fine before and now it isn't!

Stripe gives me this error:
Invalid encoding: ISO-8859-1

I have checked the Debian encoding (I think) and it says:

~# echo $LANG
C.UTF-8

Here is a very cut-down version of my code to demonstrate the problem:

#!/usr/bin/perl

use CGI::Carp qw(fatalsToBrowser);

use strict;
use warnings;
use utf8;

use JSON;

print "Content-type: application/json; charset=UTF-8\n\n";
print encode_json {
    'testing' => 'some test',
};

If I call the webhook endpoint in browser, I get the expected JSON output but Stripe gives me the encoding error.

Given that only two things have changed, server OS and Perl version, it must be one of these. Is there anything I need to do on the server to ensure that it is serving UTF-8 output correctly?

There are other problems that I think are unconnected, but wise monks may find helpful as they could have the same cause...

• .htaccess files don't work - they cause a 403 error as soon as RewriteEngine is enabled
• When I edit some (but not all) Perl scripts, the permissions change from the Plesk psacln group and the correct owner to root
• When duplicating or copying a Perl script the permissions always change to root and from 755 to 644
• If I set up FTP accounts, they only have read permission regardless of what group(s) the account gets put in

[Corion]

The error is likely not from the JSON but from the headers you resp. your webserver output.

Inspect these with (for example) curl:

curl -v https://your.server/webhook.cgi

[Bod]

Thanks...I've tracked the error down - see Re^2: JSON UTF-8

But I've learnt that I can use curl on Windows and that I can easily look at headers :)

[Bod]

I don't really know what I am looking at but my untrained eye doesn't spot anything untoward in the reply from curl...

C:\Users\User>curl -v https://test.ontheradar.uk/api/stripe/test.pl
*   Trying 194.164.26.100:443...
* Connected to test.ontheradar.uk (194.164.26.100) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET /api/stripe/test.pl HTTP/1.1
> Host: test.ontheradar.uk
> User-Agent: curl/8.4.0
> Accept: */*
>
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 09 May 2024 14:18:17 GMT
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PleskLin
<
{"testing":"some test"}* Connection #0 to host test.ontheradar.uk left
+ intact

[hippo]

print "Content-type: application/json; charset=UTF-8\n\n";

FWIW, I never specify the charset= part when sending JSON because the RFC already specifies it must be UTF-8.

Here is a very cut-down version of my code to demonstrate the problem

Looks OK otherwise. Is this an actual data set you have tested with the integration?

[Bod]

Looks OK otherwise

Thank you hippo for that comment...

Your faith in the integrity of it made me go and look again at the endpoint and the settings in Stripe. I found the error which is not what Stripe was making obvious. Digging a bit deeper I found that Stripe was also saying there was 302 redirection. The webhook endpoint in Stripe was subtly wrong causing a redirect which Stripe treats as an error.

Clearly a "senior moment" on my part

[Bod]

FWIW, I never specify the charset= part when sending JSON

Niether do I!
That was added after I got the error from Stripe...