How to get a better error message from LWP::UserAgent on missing SSL certificates?

[moritz]

I just spent several hours debugging some code using LWP::UserAgent. The code could be reduced to this:

use strict;
use warnings;
use LWP::UserAgent;

my $URL = 'https://myhost.local/';

my $ua = LWP::UserAgent->new;
$ua->ssl_opts( verify_hostname => 0 );
my $response = $ua->get($URL);

if ($response->is_success) {
    print $response->decoded_content;  # or whatever
}
else {
    die $response->status_line;
}

So pretty much a copy&paste from the LWP::UserAgent docs.

The error message I got was:

Can't connect to myhost.local:443\n\n
500 Can't connect to myhost.local:443 at foo.pl line 15.

After some debugging, strace finally showed me an ENOENT for an SSL CA file, which nudged me into the right direction.

So, my questions are:

• Why does a missing CA file cause the request to fail, even though I've used ua->ssl_opts( verify_hostname => 0 );?
• What do I have to do to get a better error message out of LWP::UserAgent? Is this a bug in LWP::UserAgent?

This is on a Debian Jessie box with perl 5.20.2, and LWP::UserAgent 6.06.

[hippo]

Why does a missing CA file cause the request to fail, even though I've used ua->ssl_opts( verify_hostname => 0 );?

Just because you aren't verifying the hostname doesn't mean it won't try to verify the certificate. The CA file is used to verify the certificate which may or may not match the hostname.

What do I have to do to get a better error message out of LWP::UserAgent?

Very difficult to get the precise problem as this is handled by the lower-level SSL modules. I'd recommend that you add

use IO::Socket::SSL 'debug4';

before you use LWP. That gives loads of output and should at least narrow down the problem to the SSL handshake if nothing more specific. I agree that this is very annoying and would be keen to see better diagnostics passed back up the tree to LWP and thus available to the users. That likely won't be easy.

[Corion]

noxxi has written some tools for checking and debugging SSL connections. In his talk about debugging SSL, he especially recommends analyze-ssl.pl.

For those who prefer spoken word over text, there also is a live talk by Steffen Ullrich on debugging SSL/TLS from the GPW 2015.

[Krambambuli]

Just to make sure: was your https server working properly otherwise, could you connect with a browser against the same URL?

[moritz]

I verified it with curl -s $URL, and it worked. Same with wget --no-check-certificate $URL.

[Krambambuli]

I've tried to reproduce the problem, but it just works here.
If I hide the CA certificate, I get an error with verify_hostname set, I don't if it is unset or set to 0.
(Perl 5.22.2, LWP::UserAgent 6.15, LWP::Protocol::https 6.06 on Fedora 24.)

[Anonymous Monk]

Dump the headers. LWP adds extra headers for more diagnostics, they start with Client

[moritz]

I did:

Client-Date: Wed, 31 Aug 2016 08:58:47 GMT
Client-Warning: Internal response

Not very helpful.