Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

parallel HTTP::Request

by RedGrinGo (Novice)
on Jan 12, 2010 at 08:58 UTC ( #816843=perlquestion: print w/replies, xml ) Need Help??
RedGrinGo has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I am trying to get response from three diff sites and to process the catenate response for some work i am doing

i really don't know what i am doing wrong Please help me

for some reason although for (1..3) the fork creating 5-6 processes and the script doe's not ends


use LWP::UserAgent; my $pid = fork(); my $resp; for (1..3) { if ($pid = fork()) { next; } if (defined $pid) { $resp .= GET(); print "=" x155 ."\n$resp\n" ."=" x155 ."\n"; exit; } else { die "Fork failed\n"; } } sub GET { my $ua = LWP::UserAgent->new; $ua->agent("Perl 5.8.8"); my $req = HTTP::Request->new("GET", ""); my $res = $ua->request($req); if ($res->is_success) { print "OK\n"; return $res->content; } else { die "HTTP::Request error"; } }

PS: You can suggest to use ParallelUA, but it's not possible in my case.

Replies are listed 'Best First'.
Re: parallel HTTP::Request
by Anonymous Monk on Jan 12, 2010 at 09:05 UTC
    Each fork gets its own copy of $resp, for forks to talk to each other you have to read/write to STDIN/STDOUT (see perlipc, perlfork). Its easier to use threads (or forks)
      #!/usr/bin/perl -- use strict; use warnings; use threads; use threads::shared; use WWW::Mechanize 1.60; #~ Main(@ARGV); Main( qw' http://ERROR/ ' ); exit(0); sub DEBUG { #~ print STDERR join ' ', threads->tid, scalar gmtime, @_, "\n"; } sub Main { my @tit; for my $url (@_) { ##!IMPORTANT! sets THREAD CONTEXT ##!! if context is void, join returns undef my $thr = threads->create( \&GET, $url ); } while ( threads->list ) { for my $thr ( threads->list ) { my $joi = $thr->join; my $err = $thr->error; push @tit, $err || $joi; } DEBUG( "threads::running", 0 + threads::running ); DEBUG( "threads::joinable", 0 + threads::joinable ); sleep 1; } ## end while ( threads->list ) print join "\n", @tit, ( time - $^T ) . " seconds\n"; } ## end sub Main sub GET { my ($url) = shift; DEBUG($url); my $ua = WWW::Mechanize->new( autocheck => 1 ); $ua->get($url); DEBUG( $url, $ua->title ); return $ua->title; } ## end sub GET __END__

      First Thank you, I will try to work with threads this is should work.

      the printing to STDOUT will be exact 3 response?

        nope printing to STDOUT is still doesn't answering the need for 3 response!

Re: parallel HTTP::Request
by happy.barney (Friar) on Jan 12, 2010 at 13:31 UTC
    line my $pid = fork(); caused that you have 6 processes.
Re: parallel HTTP::Request
by suaveant (Parson) on Jan 12, 2010 at 14:17 UTC
    aside from the my $pid = fork();

    if ($pid = fork()) { next; }

    needs to be

    if ($pid = fork()) { last; }

    By doing next you just skip an iteration, you don't actually leave the loop.

                    - Ant
                    - Some of my best work - (1 2 3)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://816843]
Approved by Corion
Front-paged by matze77
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (4)
As of 2018-06-22 04:51 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (121 votes). Check out past polls.