Re^13: Async DNS with LWP

[BrowserUk]

I'm sorry, but worrying about async DNS at this point is ... well, pointless. Let's do some math.

With your current setup, 90e6 sites with say an average of 100k per home page(*). To download that lot in your 100 hour allocation, you'd need to be fetching constantly at a rate of 25Mbytes/s. That would would (conservatively) require a 250Mbps connection. To do it in your target 3 hours you'd need an 8 Gbps connection.

Now, I'm not sure what data rates you can achieve with GSM (GPRS,EGPRS) in the US, but I'm pretty sure they'll be measured in 10s of Kbps. Not Mbps much less Gbps.

Even once you moved to your hoster, if you could sustain their 100Mbps burst rate indefinitely, 90e6 * 100k would take ~250 hours to download. And they'd cut you off long before that.

Worrying about shaving a few milliseconds here and there using asynchronous DNS is just a drop in the ocean.

(*)They seem to range from the minimalist google at 8k, up to the commercial bloat of sky.com at 250k; but 100k is a good average of the few I looked at.

[jc]

You raise some good points. What kind of throughput have you managed to implement with your solution? What bandwidth was available? I'm guessing you never saturated it, right?

[BrowserUk]

We had an early 4-cpu (real cpus not cores) SMP box with a (shared) 1Gbps link direct to the (a) backbone. We easily saturated that with 32 threads running bog standard LWP & Digest::MD5--provided we didn't store the data to disk. Even with raided (5 I think) disks, the bottleneck was storing what we could read. That was circa 7 years ago.

To do the job properly at the scale you are talking about, you'd need to run a distributed crawler, each node with dedicated, high-speed, raided local drives--or hugely expensive SSD arrays, and a distributed queueing mechanism.

[jc]

With 32 synchronous LWP sessions you saturated 1Gbps bandwith? Am i understanding you right here? A 1 Gbps connection was saturated by downloading only 32 web pages at a time? What tools did you use to monitor bandwidth consumption?