Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

WWW::Mechanize giving GET Errors

by bluesplay106 (Novice)
on Jun 13, 2013 at 15:07 UTC ( #1038768=perlquestion: print w/replies, xml ) Need Help??
bluesplay106 has asked for the wisdom of the Perl Monks concerning the following question:

So I'm trying to create a web crawler and for some reason Mechanize is giving me some weird errors. So when I run my crawler it's fine, but then it just starts giving me GET errors for every link. I tried validating the link on my browser and using $mech->get and I received no error. Does Mechanize or the website I'm crawling have some sort of search limit? Thanks

Replies are listed 'Best First'.
Re: WWW::Mechanize giving GET Errors
by kcott (Chancellor) on Jun 13, 2013 at 16:19 UTC

    G'day bluesplay106,

    "So I'm trying to create a web crawler and for some reason Mechanize is giving me some weird errors. So when I run my crawler it's fine, but then it just starts giving me GET errors for every link. I tried validating the link on my browser and using $mech->get and I received no error. Does Mechanize or the website I'm crawling have some sort of search limit? Thanks"

    All the psychic monks are on a retreat this month. If you'd like an answer before their return, you'll need to provide some additional information:

    • What are the errors?
    • What code generates the errors?
    • Under what conditions does it run fine?
    • Under what conditions does it return errors?
    • What website are you crawling?

    See: How do I post a question effectively?

    -- Ken

Re: WWW::Mechanize giving GET Errors
by Anonymous Monk on Jun 13, 2013 at 16:18 UTC
    websites are free to be whatever they want, so if they want to block you, they'll block you, thats life

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038768]
Approved by sundialsvc4
help
Chatterbox?
[marioroy]: The MCE::Hobo::Pool module will come later.
[Corion]: marioroy: Oh, that's always cool, having API-compatible modules. This makes testing and comparing things much easier
[marioroy]: IPC in MCE::Shared can handle 400k (sends) per second. That's seems a lot for being a pure-Perl module. After making the release, will come back and post a solution for a node by a fellow wanting faster logging.
[Corion]: While working on WWW::Mechanize:: Chrome, I had the suspicion that AnyEvent was doing something wrong, but I was able to swap it out for Mojolicious and the error persisted.
[Corion]: Of course, the error was in my own code ;)

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2017-05-26 08:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?