Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: LWP Posting login/redirect

by elwarren (Curate)
on Oct 27, 2004 at 00:08 UTC ( #402857=note: print w/ replies, xml ) Need Help??


in reply to LWP Posting login/redirect

LWP will by default follow redirects between 10-15 times before giving up. It is a configurable parameter. Some of the more intelligent sites will also ignore you if your UserAgent isn't something well known like Mozilla or IE. Change the default UA string to avoid discrimination. Finally, some more intelligent (or not) sites will check your referer to try and avoid deep linking or (again) scripted webcrawlers.

Lucky for us, all of these defenses can be overcome with LWP. Long live Perl! The swiss-army chainsaw!


Comment on Re: LWP Posting login/redirect

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://402857]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (16)
As of 2015-07-29 12:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (263 votes), past polls