Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re: A regex on the same content fails and works, with conditions

by hacker (Priest)
on Oct 22, 2007 at 16:29 UTC ( #646486=note: print w/replies, xml ) Need Help??

in reply to A regex on the same content fails and works, with conditions

I managed to solve this using some HTML::Element fu:

(my $author) = map $_->as_text, $t->look_down(_tag => 'a', href => +qr{^http://news\.example\.com/\?author=});

But while testing this, it appears the upstream site has some blocking/throttling mechanisms, so now I can't test it because they're throwing back pages indicating I'm "reading articles faster than a human can read" (my code had a 10-second delay in it).

Now I'm adding randomization across an array of anonymous proxies to try to alleviate that blocking, but the list of proxies is not reliable.

Too many yaks to shave in one day.

Replies are listed 'Best First'.
Re^2: A regex on the same content fails and works, with conditions (rude)
by tye (Sage) on Oct 22, 2007 at 17:06 UTC

    When a site tells you that you are hitting it too hard, it is pretty darn rude to try to thwart them by going through anonymous proxies. Instead of wasting your time trying to violate their attempts to control access to their site, why don't you just reduce the frequency of your hammering them while you compose a polite letter asking for permission (if the second step is even required).

    - tye        

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://646486]
[marto]: [id://1196212
[marto]: err same old song and dance I'm afraid
[erix]: I recognise the makings of a fine argument
[LanX]: lanx wonders ... how likely is it to talk >95% BS without intention?
[erix]: "gigantic amounts of data" is also not SQLite (imho)
talexb wonders why sqlite2 was deprecated in favour of sqlite3.

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (11)
As of 2017-07-28 15:30 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (431 votes). Check out past polls.