Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

beyond LWP return code

by Anonymous Monk
on Oct 19, 2003 at 21:40 UTC ( #300423=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks, Seekers, and Browsers.

When using LWP::UserAgent to access web sites, I note that it tries very hard to make things appear simple. Things are often not, however, actually simple. For instance: If the host returns a 500 error, LWP returns a 500 error to the caller. If the host doesn't answer, LWP returns a 500 error. This is not the truth. If you can't contact the host, you can't get an error code, including 500. If the host has no DNS entry, you get a 500 error. If the network is down, you get a 500 error. So you see where I am going. If you need to know what really happened, you have to follow-up every 500 error with some kind of troubleshooting. Same with a 400 error. If LWP doesn't like the URL, you get a 400 error just as if you had really contacted the host.

It occurs to me that LWP has to go through all the steps of contacting DNS, contacting the host, etc. So LWP must discover whether DNS is reachable, and whether there is a record for the host, and whether the host was actually contacted. It knows, but it won't say. I have expended a lot of time going through documentation (erm...) and code (yikes! The swamp is eating me alive!) at CPAN and can't find a solution.

So I ask you: Is there any way to get LWP to give up the information?

Replies are listed 'Best First'.
Re: beyond LWP return code
by pg (Canon) on Oct 19, 2003 at 22:52 UTC

    Tested and looked into the code for LWP::UserAgent. You are right, it does dictate the return code.

    I agree with you, this is wrong, as it quietly modifies the true meaning of the response code.

    Strictly speaking, in case that the host cannot be contacted or there is no response, undef shall be returned instead of a HTTP::Response object. A seperate field shall be used to give the reason why undef is returned.

    But LWP::UserAgent does give you a hint in its _msg field, which you can use:
    require LWP::UserAgent; use Data::Dumper; use strict; use warnings; my $res = LWP::UserAgent->new()->get('http://www.hopingthereisnosuchth'); print "return code = $res->{'_rc'}\n"; print "msg = $res->{'_msg'}\n";
    Under the above case, you get:
    return code = 500 msg = Can't connect to (Bad hostna +me '')

    By the way, it is better not to use the internal field names like _rc and _msg. HTTP::Response has two methods defined to return those two fields, so just call code() and message().

      Bingo! The _msg field has clues (apparently sufficient!) to decipher what happened to a request, when combined with the knowledge of whether or not the request timed out. The secrets of the HTTP::response object can be revealed with:
      my $x = $ua->simple_request($req); foreach $k (keys(%$x)) {print "$k = $x->{$k}\n";}
      Is there definitive documentation on the message meanings? I cannot locate the source to HTTP::Protocol::http at Reverse engineering is going pretty well, but that's not the preferred method, now is it?
Re: beyond LWP return code
by simonm (Vicar) on Oct 19, 2003 at 22:53 UTC
    It's not entirely clear what you're asking for, but looking at the source of LWP::UserAgent, it seems like you can check the response headers to see if they include a "Client-Warning: Internal response" line -- if so, then it's an LWP-generated "fake" error. (This is defined in sub _new_response, near the end of LWP::UserAgent.)

      If you look at all the possible values LWP::UserAgent assigns to Client-Warning field, you will realize that it is not really related to what the original post asked.

      • Once it says "redirect loop detected";
      • rest of times are all related to authentication.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://300423]
Approved by blokhead
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2017-06-24 11:00 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (557 votes). Check out past polls.