Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: check multiple URLS in the same time

by thonar (Monk)
on Aug 23, 2012 at 11:10 UTC ( #989283=note: print w/ replies, xml ) Need Help??


in reply to check multiple URLS in the same time

I picked the idea of zentara using Parallel::ForkManager and LWP::UserAgent, here ist what works for me:

#!/usr/bin/perl -w use strict; use warnings; use Parallel::ForkManager; use LWP::UserAgent ; my $refUrlList = []; my $maxChild = "3"; my $maxRedir = "5"; my $maxSize = "300"; my $timeout = "5"; (my $logFile = $0) =~ s/\.(\w+)$/\.log/; for ( @ARGV){ if ( -f ){ open ( IN, "<", $_) or die "Cant open $_: $!\n"; while (<IN>){ push(@$refUrlList, $_); } close ( IN) or die "Close Failed: $_: $!\n"; } } + + &multiUrl( $refUrlList); sub multiUrl { my $refUrls = shift; open ( OUT, ">> $logFile") or die "Can't open $logFile: $!\n"; my $pm = new Parallel::ForkManager( $maxChild); for ( @$refUrls) { chomp( my $url = $_); $pm->start and next; my $ua = LWP::UserAgent->new; $ua->agent( $0); $ua->max_redirect( $maxRedir); $ua->max_size($maxSize); $ua->timeout( $timeout); my $res = $ua->get( $url); print OUT "$url | " . $res->code . " | " . $res->message ."\n"; print "$url | " . $res->code . " | " . $res->message ."\n"; $pm->finish; } $pm->wait_all_childs; close ( OUT) or die "Close Failed: $logFile: $!\n"; }


Comment on Re: check multiple URLS in the same time
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://989283]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2014-09-20 06:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (155 votes), past polls