Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Mechanize, Javascript, Cookies, and you!

by aquarium (Curate)
on Jun 21, 2011 at 04:25 UTC ( #910668=note: print w/ replies, xml ) Need Help??


in reply to Mechanize, Javascript, Cookies, and you!

maybe jmeter could do it or something like serverside javascript like rhino...but seems you're heading down a slippery slope anyway. make sure you setup your http headers and response codes to look like a browser. but as per my slippery slope observation, these kinds of systems are also usually rigged with a bit of random checks anyway with captcha showing up. in the event someone is watching and doesn't appreciate your automation attempts, they may even start popping up more captchas based on your ip address etc., or could start popping up captchas incessantly if they're really against this...and thus defeat all your coding powers at press of a button.

the hardest line to type correctly is: stty erase ^H


Comment on Re: Mechanize, Javascript, Cookies, and you!
Re^2: Mechanize, Javascript, Cookies, and you!
by GaijinPunch (Pilgrim) on Jun 21, 2011 at 05:21 UTC
    Indeed, advice to be considered. All in all, yes, I think they are against it, but not going to risk pissing of everyone. Without giving away too much, as my scraper translates for those who otherwise not be a customer, it's in their best interest to allow it. However, they've also gotta think about malicious scrapers. I'll bet anyone a shiny quarter the beef up in security is in response to Sony getting sodomized recently.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://910668]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2014-04-21 01:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (489 votes), past polls