|
|
|
| We don't bite newbies here... much | |
| PerlMonks |
Re^3: Advice on Efficient Large-scale Web Crawlingby Scott7477 (Chaplain) |
| on May 07, 2006 at 06:54 UTC ( [id://547868]=note: print w/replies, xml ) | Need Help?? |
|
According to Google API a license key only allows for 1,000 automated queries per day. This page while somewhat dated, provides some data relevant to this discussion. A couple of key points from that data include: -Netcraft estimated that 42.8 million web servers existed. Assuming 50 URLs per web server gives over 2.1 billion URLs. If the OP is randomly selecting URLs the chances of any particular server being significantly inconvenienced are small, in my estimation.
In Section
Seekers of Perl Wisdom
|
|
||||||||||||||||||||||