Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

WWW::Mechanize giving GET Errors

by bluesplay106 (Novice)
on Jun 13, 2013 at 15:07 UTC ( #1038768=perlquestion: print w/ replies, xml ) Need Help??
bluesplay106 has asked for the wisdom of the Perl Monks concerning the following question:

So I'm trying to create a web crawler and for some reason Mechanize is giving me some weird errors. So when I run my crawler it's fine, but then it just starts giving me GET errors for every link. I tried validating the link on my browser and using $mech->get and I received no error. Does Mechanize or the website I'm crawling have some sort of search limit? Thanks

Comment on WWW::Mechanize giving GET Errors
Replies are listed 'Best First'.
Re: WWW::Mechanize giving GET Errors
by kcott (Canon) on Jun 13, 2013 at 16:19 UTC

    G'day bluesplay106,

    "So I'm trying to create a web crawler and for some reason Mechanize is giving me some weird errors. So when I run my crawler it's fine, but then it just starts giving me GET errors for every link. I tried validating the link on my browser and using $mech->get and I received no error. Does Mechanize or the website I'm crawling have some sort of search limit? Thanks"

    All the psychic monks are on a retreat this month. If you'd like an answer before their return, you'll need to provide some additional information:

    • What are the errors?
    • What code generates the errors?
    • Under what conditions does it run fine?
    • Under what conditions does it return errors?
    • What website are you crawling?

    See: How do I post a question effectively?

    -- Ken

Re: WWW::Mechanize giving GET Errors
by Anonymous Monk on Jun 13, 2013 at 16:18 UTC
    websites are free to be whatever they want, so if they want to block you, they'll block you, thats life

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038768]
Approved by sundialsvc4
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (7)
As of 2015-09-01 21:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred temperature scale is:










    Results (49 votes), past polls