Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Any idea why LWP:Simple doesn't like this particular website?

by Cody Pendant (Prior)
on Jul 02, 2001 at 03:31 UTC ( #93078=perlquestion: print w/ replies, xml ) Need Help??
Cody Pendant has asked for the wisdom of the Perl Monks concerning the following question:

I have a script which checks various websites to see if they're updated, using LWP:Simple, and it seems to hang/time out when going to this site http://www.theredkitchen.net/

can anyone tell me why?

I'm assuming there's something about its server that is causing LWP:Simple to fail to access it, but it's easily accessible with my regular graphic browsers, so it's not that the site is down -- LWP happily and immediately returns an error for sites that are down, in my experience.

Comment on Any idea why LWP:Simple doesn't like this particular website?
Re: Any idea why LWP:Simple doesn't like this particular website?
by epoptai (Curate) on Jul 02, 2001 at 03:39 UTC
    Update: Tried lwp on that site and it works.
    perl -e "use LWP::Simple; getprint 'http://www.theredkitchen.net/'"

    It's possible for a site to block lwp transfers via $ENV{'HTTP_USER_AGENT'} on the server side. You could try using LWP::UserAgent to specify your own agent:

    $ua = LWP::UserAgent->new; $ua->agent('Odyssey/2001');

    --
    Check out my Perlmonks Related Scripts like framechat, reputer, and xNN.

      OK I'm using the following script to do it, and it returns absolutely nothing except "The script has successfully started up.":
      #!/usr/bin/perl -w $| = 1; use diagnostics; use CGI::Carp qw(fatalsToBrowser); use LWP::Simple; print "Content-type: text/html\n\n"; print "The script has successfully started up.\n\n<BR><BR>"; # just to + prove there's nothing wrong $theURL = 'http://www.theredkitchen.net/'; unless($doc=get($theURL)){ print "can't get the URL"; die "$!"; } print $doc; exit;
        As long as you've got -w and diagnostics you might as well use strict and put them to good use. Also declare your variables with my so typos will be apparent. When that is done the only error left is this enigmatic line:
        die "$!";
        which should just be exit so it won't try to print the empty $doc and get that nasty uninitialized value warning. Try this:
        #!/usr/bin/perl -w use strict; # added use diagnostics; use CGI::Carp qw(fatalsToBrowser); use LWP::Simple; my$doc; my$theURL = 'http://www.theredkitchen.net/'; print "Content-type: text/html\n\n"; print "The script has successfully started up.\n\n<BR><BR>"; unless($doc=get($theURL)){ print "can't get the URL"; exit; # added } print $doc; exit;
        This code is tested and works. If it doesn't work for you the problem may lie elsewhere...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://93078]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (13)
As of 2014-12-19 14:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (84 votes), past polls