Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re: Fastest way to download many web pages in one go?

by sundialsvc4 (Abbot)
on Oct 15, 2013 at 14:09 UTC ( #1058301=note: print w/replies, xml ) Need Help??

in reply to Fastest way to download many web pages in one go?

As an aside, at one “shop” where I was working, they had a variation of the Unix execargs command which supported pooling.   It was just an -n number_of_children parameter (or something like that ...), but it sure was useful.   The command worked in the usual way ... read lines from STDIN and execute a command with that line in its argument-string ... but it supported n children doing the commands simultaneously.   Each child ran, did its thing, and then died.   Maybe this is a standard feature ... I don't know ... but it cropped up everywhere in the stuff that they were doing, as a useful generalization.   Here, you’d feed it a file containing a list of URLs and use it to drive a command that took one URL as a command-line parameter, retrieved and processed it.   Since each process would actually spend most of its time waiting for some host to respond, you could run a very large number of ’em.

  • Comment on Re: Fastest way to download many web pages in one go?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1058301]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (8)
As of 2017-09-21 12:51 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (247 votes). Check out past polls.