Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: Fastest way to download many web pages in one go?

by sundialsvc4 (Abbot)
on Oct 15, 2013 at 14:09 UTC ( #1058301=note: print w/replies, xml ) Need Help??

in reply to Fastest way to download many web pages in one go?

As an aside, at one “shop” where I was working, they had a variation of the Unix execargs command which supported pooling.   It was just an -n number_of_children parameter (or something like that ...), but it sure was useful.   The command worked in the usual way ... read lines from STDIN and execute a command with that line in its argument-string ... but it supported n children doing the commands simultaneously.   Each child ran, did its thing, and then died.   Maybe this is a standard feature ... I don't know ... but it cropped up everywhere in the stuff that they were doing, as a useful generalization.   Here, you’d feed it a file containing a list of URLs and use it to drive a command that took one URL as a command-line parameter, retrieved and processed it.   Since each process would actually spend most of its time waiting for some host to respond, you could run a very large number of ’em.

  • Comment on Re: Fastest way to download many web pages in one go?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1058301]
and not a whimper to be heard...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (7)
As of 2017-11-19 16:36 GMT
Find Nodes?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:

    Results (282 votes). Check out past polls.