Request by LWP Useragent refused by the web server but not by others

[epoch4life]

I have in the past successfully used LWP::Useragent to crawl web sites in http or https to download data, but just can't make it work anymore with this new server. I feel this might be due to the server trying to refuse useragents deliberately. I read some post in this thread which suggested to impersonate a Firefox request. I tried my best in this regard all to no avail.

More specifically, I'm trying to use LWP useragent to automate the collection of data from a site, but was always get refused with this message

500 Can't connect to tutorialregistration.uws.edu.au:443 (SSL connect 
+attempt failed because of handshake problemserror:00000000:lib(0):fun
+c(0):reason(0))

I have narrowed it down to accessing just the URL https://tutorialregistration.uws.edu.au/aplus/admin/adminLogin.do which I can directly access from a browser, but failed with the above message when using LWP Useragent.

This can be shown via

perl -MLWP::Simple -e "getprint 'https://tutorialregistration.uws.edu.
+au/aplus/admin/adminLogin.do'"

or

use LWP::UserAgent;
$ua = new LWP::UserAgent;

$req = new HTTP::Request 'GET' =>
'https://tutorialregistration.uws.edu.au/aplus/admin/adminLogin.do';

# impersonate a firefox brower  
$ua->agent("Mozilla/5.0 (Windows NT 6.1; rv:29.0) Gecko/20100101 Firef
+ox/29.0");
$req->header(
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*
+;q=0.8',
'Accept-Language' => 'en-US,en;q=0.5',
'Accept-Encoding' => 'gzip, deflate',
'Cookie' => '',
'Referer' => 'https://www.uws.edu.au/',
'Connection' => 'keep-alive',  
);

$res = $ua->request($req);
print "content-type:text/html\n\n";
print $res->content;

In both cases, if I replace the webpage URL by another https page (inside or outside Intranet), they both work fine. I really can't figure out what has gone wrong here. Please help. Many thanks.

David

[Khen1950fx]

Don't forget to use strictures:

use strict;
use warnings;

You can see what the handshake is doing by using IO::Socket::SSL.

#!/usr/bin/perl

use strict;
use warnings;
use IO::Socket::SSL qw(debug3);

require LWP::UserAgent;
require HTTP::Request;

my $ua = LWP::UserAgent->new;
$ua->agent(
    "Mozilla/5.0 (Windows NT 6.1; rv:29.0) Gecko/20100101 Firefox/29.0
+");
$ua->timeout(10);
$ua->protocols_allowed(['https']);

my $req = new HTTP::Request 'GET' =>
    'https://tutorialregistration.uws.edu.au/aplus/admin/adminLogin.do
+';
$req->header(
    'Accept' =>
        'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0
+.8',
    'Accept-Language' => 'en-US,en;q=0.5',
    'Accept-Encoding' => 'gzip, deflate',
    'Cookie'          => '',
    'Referer'         => 'https://www.uws.edu.au/',
    'Connection'      => 'keep-alive',
);

my $res = $ua->request($req);
print "content-type:text/html\n\n";
print $res->content;

[epoch4life]

Many thanks to all the helpers. It looks like there are some PERL configuration problems, because I now realised that the script does work on my laptop's PERL. I'll get my system admin to fix the PERL configuration.

[Anonymous Monk]

Re^5: HTTP error response code 500 using LWP::UserAgent on one site, but not on any other says

use IO::Socket::SSL qw(debug3);

Also, http://search.cpan.org/perldoc/Crypt::SSLeay#SSL_diagnostics_and_Debugging lists $ENV{HTTPS_DEBUG} = 1;

among other tips

[sundialsvc4]

Yep, “and the good news is,” a message such as SSL handshake problem tells you definitively that the SSL connection is never successfully getting established ... so, the root cause of the problem is not as you fear.   Your fears in this case are a red-herring.   The server isn’t refusing you.   It never hears from you at all.   Follow the links mentioned above to help resolve the problem.   (And, if you have administrative access to the server that you are attempting to use, always check its server logs.   Your own computer may also log failed connection-attempts and provide useful diagnostic information.)

[epoch4life]

Thanks, and it turns out you are absolutely right.