Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

parallel HTTP::Request

by RedGrinGo (Novice)
on Jan 12, 2010 at 08:58 UTC ( #816843=perlquestion: print w/ replies, xml ) Need Help??
RedGrinGo has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I am trying to get response from three diff sites and to process the catenate response for some work i am doing

i really don't know what i am doing wrong Please help me

for some reason although for (1..3) the fork creating 5-6 processes and the script doe's not ends

WHAT DO I DO WRONG?

use LWP::UserAgent; my $pid = fork(); my $resp; for (1..3) { if ($pid = fork()) { next; } if (defined $pid) { $resp .= GET(); print "=" x155 ."\n$resp\n" ."=" x155 ."\n"; exit; } else { die "Fork failed\n"; } } sub GET { my $ua = LWP::UserAgent->new; $ua->agent("Perl 5.8.8"); my $req = HTTP::Request->new("GET", "http://www.google.com/"); my $res = $ua->request($req); if ($res->is_success) { print "OK\n"; return $res->content; } else { die "HTTP::Request error"; } }

PS: You can suggest to use ParallelUA, but it's not possible in my case.

Comment on parallel HTTP::Request
Download Code
Re: parallel HTTP::Request
by Anonymous Monk on Jan 12, 2010 at 09:05 UTC
    Each fork gets its own copy of $resp, for forks to talk to each other you have to read/write to STDIN/STDOUT (see perlipc, perlfork). Its easier to use threads (or forks)

      First Thank you, I will try to work with threads this is should work.

      the printing to STDOUT will be exact 3 response?

        nope printing to STDOUT is still doesn't answering the need for 3 response!

      #!/usr/bin/perl -- use strict; use warnings; use threads; use threads::shared; use WWW::Mechanize 1.60; #~ Main(@ARGV); Main( qw' http://yahoo.com/ http://google.com/ http://cpan.org/ http://ERROR/ ' ); exit(0); sub DEBUG { #~ print STDERR join ' ', threads->tid, scalar gmtime, @_, "\n"; } sub Main { my @tit; for my $url (@_) { ##!IMPORTANT! sets THREAD CONTEXT ##!! if context is void, join returns undef my $thr = threads->create( \&GET, $url ); } while ( threads->list ) { for my $thr ( threads->list ) { my $joi = $thr->join; my $err = $thr->error; push @tit, $err || $joi; } DEBUG( "threads::running", 0 + threads::running ); DEBUG( "threads::joinable", 0 + threads::joinable ); sleep 1; } ## end while ( threads->list ) print join "\n", @tit, ( time - $^T ) . " seconds\n"; } ## end sub Main sub GET { my ($url) = shift; DEBUG($url); my $ua = WWW::Mechanize->new( autocheck => 1 ); $ua->get($url); DEBUG( $url, $ua->title ); return $ua->title; } ## end sub GET __END__
Re: parallel HTTP::Request
by happy.barney (Pilgrim) on Jan 12, 2010 at 13:31 UTC
    line my $pid = fork(); caused that you have 6 processes.
Re: parallel HTTP::Request
by suaveant (Parson) on Jan 12, 2010 at 14:17 UTC
    aside from the my $pid = fork();

    if ($pid = fork()) { next; }

    needs to be

    if ($pid = fork()) { last; }

    By doing next you just skip an iteration, you don't actually leave the loop.

                    - Ant
                    - Some of my best work - (1 2 3)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://816843]
Approved by Corion
Front-paged by matze77
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (14)
As of 2014-09-19 16:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (143 votes), past polls