Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

LWP - 500 Internal Error

by oalvi (Initiate)
on Sep 01, 2009 at 12:20 UTC ( #792626=perlquestion: print w/ replies, xml ) Need Help??
oalvi has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have the following code which I'm using for monitoring URLs:
# !C:\Perl\bin use strict; use warnings; use LWP::UserAgent; #my $wsr = LWP::UserAgent -> new; #$wsr -> timeout( 20 ); my $url = shift or die "URL expected\n"; my $useragent = LWP::UserAgent->new; $useragent->proxy(['http', 'ftp','https'], 'http://192.168.5.5:8080/') +; $useragent -> timeout( 20 ); my $request = HTTP::Request->new( HEAD => $url ); my $response = $useragent->request($request); print $response->status_line, "\n";

It works fine for most URLs, http or https. However for some URLs, which function fine from the explorer, I get errors as follows:

https://elmag.elm.com.sa

500 Internal Server Error

https://www.etv.com.sa

500 Internal Server Error

Other https addresses work fine, for example

https://eserve.com.sa

200 OK

Looking forward to your support and assistance.

Comment on LWP - 500 Internal Error
Download Code
Re: LWP - 500 Internal Error
by Anonymous Monk on Sep 01, 2009 at 12:29 UTC
      Hi and Thank for the link, Was my thread, vry interesting too - but i didn't think I could start from there anew. Anyway, I tried adding the sfprint debug, doesn't work for me yet.
      # !C:\Perl\bin use strict; use warnings; use LWP::UserAgent; @ARGV = qw[ http://www.eserve.com.sa/ ]; my $url = shift or die "URL expected\n"; my $useragent = LWP::UserAgent->new; $useragent->proxy(['http', 'ftp','https'], 'http://192.168.5.5:8080/') +; $useragent -> timeout( 20 ); my $request = HTTP::Request->new( GET => $url ); my $response = $useragent->request($request); my $err = sprintf "(%d)(%s)(%d)(%s)\n", $!,$!,$^E,$^E; print $useragent->proxy('http'),"\n"; print $response->status_line, "\n"; print "Err : $err \n";

      I get this error

      http://192.168.5.5:8080/ 200 OK Err : (9)(Bad file descriptor)(0)()

      Seems it ran everything alrights, something simple here I must be missing

        200 OK means there was no error
Re: LWP - 500 Internal Error
by ig (Vicar) on Sep 01, 2009 at 14:18 UTC

    Can you review the server error logs to find the causes of the errors?

    You can use a network sniffer (e.g. wireshark) to capture the traffic between your system and the server. You might find some clues by comparing successful and failed accesses. Even with HTTPS, not everything is encrypted.

    There are tools that might allow you to inspect the HTTPS exchange with the server. If Firefox browser works, you might be able to use some of its debugging features to find out what is happening. Otherwise, there are proxy servers which might allow you to inspect the exchange (e.g. fiddler. Ultimately, you could brute force decrypt the traffic, thought this might take some time.

    Might cookies be required by the server?

Re: LWP - 500 Internal Error
by Anonymous Monk on Jul 07, 2010 at 08:33 UTC
    You should set 'agent' parameter for your userAgent object. It seems that some sites verify user_agent field in Request and reject connection if it is empty or differs from common variants like 'mozilla', 'opera', etc >> $useragent -> agent( 'Mozilla' );

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://792626]
Approved by tbone1
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (10)
As of 2014-07-25 07:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (169 votes), past polls