Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

parallel HTTP::Request

by RedGrinGo (Novice)
on Jan 12, 2010 at 08:58 UTC ( #816843=perlquestion: print w/ replies, xml ) Need Help??
RedGrinGo has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I am trying to get response from three diff sites and to process the catenate response for some work i am doing

i really don't know what i am doing wrong Please help me

for some reason although for (1..3) the fork creating 5-6 processes and the script doe's not ends

WHAT DO I DO WRONG?

use LWP::UserAgent; my $pid = fork(); my $resp; for (1..3) { if ($pid = fork()) { next; } if (defined $pid) { $resp .= GET(); print "=" x155 ."\n$resp\n" ."=" x155 ."\n"; exit; } else { die "Fork failed\n"; } } sub GET { my $ua = LWP::UserAgent->new; $ua->agent("Perl 5.8.8"); my $req = HTTP::Request->new("GET", "http://www.google.com/"); my $res = $ua->request($req); if ($res->is_success) { print "OK\n"; return $res->content; } else { die "HTTP::Request error"; } }

PS: You can suggest to use ParallelUA, but it's not possible in my case.

Comment on parallel HTTP::Request
Download Code
Re: parallel HTTP::Request
by Anonymous Monk on Jan 12, 2010 at 09:05 UTC
    Each fork gets its own copy of $resp, for forks to talk to each other you have to read/write to STDIN/STDOUT (see perlipc, perlfork). Its easier to use threads (or forks)

      First Thank you, I will try to work with threads this is should work.

      the printing to STDOUT will be exact 3 response?

        nope printing to STDOUT is still doesn't answering the need for 3 response!

      #!/usr/bin/perl -- use strict; use warnings; use threads; use threads::shared; use WWW::Mechanize 1.60; #~ Main(@ARGV); Main( qw' http://yahoo.com/ http://google.com/ http://cpan.org/ http://ERROR/ ' ); exit(0); sub DEBUG { #~ print STDERR join ' ', threads->tid, scalar gmtime, @_, "\n"; } sub Main { my @tit; for my $url (@_) { ##!IMPORTANT! sets THREAD CONTEXT ##!! if context is void, join returns undef my $thr = threads->create( \&GET, $url ); } while ( threads->list ) { for my $thr ( threads->list ) { my $joi = $thr->join; my $err = $thr->error; push @tit, $err || $joi; } DEBUG( "threads::running", 0 + threads::running ); DEBUG( "threads::joinable", 0 + threads::joinable ); sleep 1; } ## end while ( threads->list ) print join "\n", @tit, ( time - $^T ) . " seconds\n"; } ## end sub Main sub GET { my ($url) = shift; DEBUG($url); my $ua = WWW::Mechanize->new( autocheck => 1 ); $ua->get($url); DEBUG( $url, $ua->title ); return $ua->title; } ## end sub GET __END__
Re: parallel HTTP::Request
by happy.barney (Pilgrim) on Jan 12, 2010 at 13:31 UTC
    line my $pid = fork(); caused that you have 6 processes.
Re: parallel HTTP::Request
by suaveant (Parson) on Jan 12, 2010 at 14:17 UTC
    aside from the my $pid = fork();

    if ($pid = fork()) { next; }

    needs to be

    if ($pid = fork()) { last; }

    By doing next you just skip an iteration, you don't actually leave the loop.

                    - Ant
                    - Some of my best work - (1 2 3)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://816843]
Approved by Corion
Front-paged by matze77
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (14)
As of 2015-07-02 13:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (41 votes), past polls