http://www.perlmonks.org?node_id=436178


in reply to What is the fastest way to download a bunch of web pages?

Maybe. There are many parameters that determine what the most efficient way is to download a number of webpages. Some of the more important issues are the capacity of your box (number of CPUs, your amount of memory, your disk I/O, your network I/O, what else is running on it), the capacity of the network between your and the servers you are downloading from, and the setup of the servers you are querying.

If you are really serious about the speed issue, you need to look at your infrastructure. All we can do here is guess, or present our own experience as the absolute truth, both not uncommon on Perl forums, but probably not very useful for anyone.