Be aware that LWP::UserAgent creates and emits its own 500 errors on certain occasions, see Re^2: HTTP error response code 500 using LWP::UserAgent on one site, but not on any other by afoken. So, what you are seeing may be coming from LWP's handling of the transaction and not the server (running your cgi-script). Two differences between Chrome and your LWP test script can be the response/connection timeout and the user-agent string, in addition to certificate-related problems which you seem to have excluded.

The intermittent behaviour you mentioned from the 2nd script can again be cause by short timeouts, e.g. when the script fetches the cached file it takes very short time and is within the timeout. But when the cache expires, it will take significantly longer and can fail the timeout. But even if that script does not use a cache, try setting a long timeout in both scripts (the cgi and the test) and see.

When I use LWP for purposes other than playing around, I want to keep as much control as possible: I always use LWP::UserAgent.

Finally, there's also LWP::Curl with same API which you can use as an alternative, though LWP::UserAgent never failed me. (I also prefer to use a host with unix CLI so as to avoid hair pulling, but I said that before already...)

bw, bliako