Where is LWP::Simple::get error information stored?

[ajam]

Hello Monks,

I wrote a short program which needed to occasionally download Wikipedia articles. To my frustration the pages were not being downloaded. Luckily, I stumbled upon LWP::Simple::get($url) does not work for some urls, which basically states the default user-agent is blocked. After altering that my program worked like a charm.

But now my question is, how can I programmatically determine errors like this from LWP? It appears nothing is stored in $! after the failed get attempt. Does LWP have its own error variable? It would be nice for a program like the following to relay the error information to the user.

use warnings;
use strict;
use LWP::Simple;

my @urls = (
    'http://www.google.es/',
    'http://www.perlmonks.org/',
    'http://es.wikipedia.org/',
);

foreach my $url (@urls) {
    my $content = get $url;
    $url =~ /\.(.*)\./;
    if ($content) {
        say "Received response from $1";
    } else {
        say "No response from $1, error ($!)";
    }
}

[BrowserUk]

how can I programmatically determine errors like this from LWP?

The LWP::Simple POD for get explains:

You will not be able to examine the response code or response headers (like 'Content-Type') when you are accessing the web using this function. If you need that information you should use the full OO interface (see the LWP::UserAgent manpage).

However, there is (kinda) a way around it. If you look at getprint(), it returns an HTTP error code, but of course, you probably want the content in your program, not sent to stdout. So, the trick I've used in the past is to open a ramfile and select it before calling getprint(). That way you get the content into a variable and the status code:

my $content;
open RAM, '>', \$content;
my $stdoutsaved = select;
select *RAM;
my $status = getprint( $url );
select $stdoutsaved;
close RAM;

## Now the status is in $status and the content in $content.

Quite why get() wasn't implement to return both:

my( $status, $content ) = get( $url );

I guess we'll never know, but it is unlikely to change now.

[ajam]

Thanks for the explanation and an example work around. For the sake of maintainability though I think I will avoid using this crafty method.

[BrowserUk]

For the sake of maintainability though I think I will avoid using this crafty method.

It is not at all clear to me why you think it need be a maintenance problem. Just wrap it over in a function something like:

sub getWithStatus {
    my $url = shift;
    my $content;
    open my $RAM, '>', \$content;
    my $stdoutsaved = select;
    select *$RAM;
    my $status = getprint( $url );
    select $stdoutsaved;
    close $RAM;
    return $status, $content;
}

...

my( $status, $content ) = getWithStatus( $url );
if( $status ne RC_OK ) {
    die "Get $url failed with HTTP status: $status";
}
...

Social/div

[Anonymous Monk]

Sadly it isn't stored anywhere, even LWP::UserAgent itself doesn't store that stuff, it forces you to store it

WWW::Mechanize on the other hand stores the responses so you don't have to, it even (optionally, by default ) throws exceptions

[ajam]

Thanks for the information. I will look into WWW::Mechanize.

[Anonymous Monk]

LWP::UserAgent doesn't?

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->env_proxy;

my $response = $ua->get('http://example.org/');

print "HTTP response code was: ", $response->status_line, "\n";

#print "Content was:\n", $response->content, "\n";

HTTP response code was: 200 OK
HTTP response code was: 404 Not Found
HTTP response code was: 500 Can't connect to 127.0.0.1:80 (Invalid arg
+ument)

[Anonymous Monk]

Of course it doesn't, it forces you to store it

my $response = $ua->get('http://example.org/');

[SergioQ]

I appear to be facing the same issue, “default user agent is blocked.” May I ask how you worked around it?

[marto]

You replied to an old post. If you're hitting a message like this it's likely you're scraping something the site owner doesn't intend to be automatically scraped. Consider contacting them, perhaps they have an API they'd prefer you use.