Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

WWW::Mechanize giving GET Errors

by bluesplay106 (Novice)
on Jun 13, 2013 at 15:07 UTC ( #1038768=perlquestion: print w/ replies, xml ) Need Help??
bluesplay106 has asked for the wisdom of the Perl Monks concerning the following question:

So I'm trying to create a web crawler and for some reason Mechanize is giving me some weird errors. So when I run my crawler it's fine, but then it just starts giving me GET errors for every link. I tried validating the link on my browser and using $mech->get and I received no error. Does Mechanize or the website I'm crawling have some sort of search limit? Thanks

Comment on WWW::Mechanize giving GET Errors
Replies are listed 'Best First'.
Re: WWW::Mechanize giving GET Errors
by kcott (Abbot) on Jun 13, 2013 at 16:19 UTC

    G'day bluesplay106,

    "So I'm trying to create a web crawler and for some reason Mechanize is giving me some weird errors. So when I run my crawler it's fine, but then it just starts giving me GET errors for every link. I tried validating the link on my browser and using $mech->get and I received no error. Does Mechanize or the website I'm crawling have some sort of search limit? Thanks"

    All the psychic monks are on a retreat this month. If you'd like an answer before their return, you'll need to provide some additional information:

    • What are the errors?
    • What code generates the errors?
    • Under what conditions does it run fine?
    • Under what conditions does it return errors?
    • What website are you crawling?

    See: How do I post a question effectively?

    -- Ken

Re: WWW::Mechanize giving GET Errors
by Anonymous Monk on Jun 13, 2013 at 16:18 UTC
    websites are free to be whatever they want, so if they want to block you, they'll block you, thats life

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038768]
Approved by sundialsvc4
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (13)
As of 2015-07-31 19:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (280 votes), past polls