It would be interesting to see how the various options stack up against each other, but they would all ned to be run from the same place to get anything like a reasonable comparison.
From where I am, it makes little or no difference to the total time whether I do all 10 in parallel or serially. The bottleneck is entirely the narrowness of my 40K pipe. YMMV.
To that end, here's a threaded solution:
#! perl -slw
use strict;
use threads;
use Thread::Queue;
use LWP::Simple;
use Time::HiRes qw[ time ];
$|=1;
our $THREADS ||= 3;
our $PATH ||= 'tmp/';
sub fetch {
my $Q = shift;
while( $Q->pending ) {
my $url = $Q->dequeue;
my $start = time;
warn "$url : "
. getstore( "http://$url/", "$PATH$url.htm" )
. "\t" .( time() - $start ) . $/;
}
}
my $start = time;
my $Q = new Thread::Queue;
$Q->enqueue( map{ chomp; $_ } <DATA> );
my @threads = map{ threads->new( \&fetch, $Q ) } 1 .. $THREADS;
$_->join for @threads;
print 'Done in: ', time - $start, ' seconds.';
__DATA__
www.google.com
www.yahoo.com
www.amazon.com
www.ebay.com
www.perlmonks.com
news.yahoo.com
news.google.com
www.msn.com
www.slashdot.org
www.indymedia.org
Examine what is said, not who speaks.
Silence betokens consent.
Love the truth but pardon error.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|