Using LWP::UserAgent to Get Web Pages

[jayanth]

Hi,

I'm using the LWP::UserAgent module to access web pages. Given below is the code which is supposed to return an HTTP::Response object with the contents of the requested page.

#!/usr/bin/perl

use LWP::UserAgent;

print "Content-type: text/html\n\n";

my $ua = new LWP::UserAgent();
my $url = 'http://www.google.com/';
my $httpresp = $ua->get($url);

print $httpresp->content;

When I run this code, I get the message '500 Can't connect to www.google.com:80 (Bad hostname 'www.google.com')'.

I'm running the code on a Fedora Core 6 machine. I access the Internet directly, without any proxy server. I'm not using any firewall either.

Please help.

Thanks,

Jayanth

[Popcorn Dave]

It works just fine on Windows too.

How many times did you try this? Did it ever return anything for you? The reason I ask is that I seem to recall that Google doesn't really like people writing their own page scrapers so they may have blocked your IP temporarily. They have their own tools they want you to use.

What you might consider trying is changing the URL that you're trying to access to see if Google is blocking your IP.

The other question I have is why you have the here doc in that snippet of code, because you're never using it. Or was it part of the larger program?

[pajout]

I ran your code on my FC6, it worked fine.

[moritz]

What happens if you start this script from the command line, not through CGI?

Can you access non-Google URLs?

Somehow this doesn't smell like a perl problem to me - but I could be wrong, of course.

[naikonta]

I'm running the code on a Fedora Core 6 machine. I access the Internet directly, without any proxy server. I'm not using any firewall either.

With such situation, a) did it work from your browser? b) can you confirm the result of the following commands?

$ HEAD http://www.google.com/

# and

$  perl -MLWP::Simple -le 'print head "http://www.google.com" ? "OK" :
+ "NO"'

[varian]

To learn more about the cause of the error the following will get you more verbose output from LWP:

use LWP::Debug qw(+);

My hinch is that this is either related to your DNS settings or to security settings on your system.

Try to 'ping' google to check your DNS settings.
If that works fine then check that the ping cmd also works in the context of a cgi environment executed by your webserver (e.g. include with backticks in a cgi perl prog).

[jayanth]

Hi,

Popcorn Dave: It doesn't work for any site.

moritz: It works if I run the script from the command line!

naikonta: Websites are accessible from the browser. The second command you suggested - 'perl -MLWP::Simple -le 'print head "http://www.google.com" ? "OK" : "NO"' - returns OK.

varian: Pinging 'www.google.com' is successful. As for doing it from the CGI environment, I have no idea how to go about it! :-( I'll check it out anyway.

Thanks for the responses. Hope someone will be able to suggest a solution.

Jayanth

[Anonymous Monk]

What about varians Debug suggestion?

[jayanth]

There was no difference in behaviour.

Jayanth