Parallel LWP usage

[metaperl]

I'm interested in posting to 5 or 6 HTTP servers concurrently and getting back responses.

This is a coarse-grained parallel operation and the LWP::ParallelUserAgent example is exactly what I need.

But I notice several unresolved bugs for this package.

Further, it has not seen an update in 5 years

So, I'm curious as to how you would handle such a functional requirement... is there a more popular module for this? One with fewer bugs and more recent releases? Do I need to write my own thang in POE similar to what merlyn shows here?

[metaperl]

Possum in #perl-help on irc.perl.org turned me on to AnyEvent::HTTP - it looks super duper!

[zwon]

++ I recently used this package, it's great. Highly recommend give it a try. Also WWW::Curl::Multi may be useful for you.

[metaperl]

well, more news. I just tried to install LWP::Parallel::UserAgent and see that it is not updated for the latest LWP and the test suite fails...

[tbrannon@devel FindALocal]$ sudo cpan LWP::Parallel::UserAgent
)

  CPAN.pm: Going to build M/MA/MARCLANG/ParallelUserAgent-2.57.tar.gz


           **** WARNING: Unsupported Version of libwww found! ****

This library only supports libwww up to version 5.76. You seem to have
version 5.821 installed, which has not yet been tested by the package 
+author
for compatibility. Some or all tests might fail!

Install at your own risk, or wait until a more recent version of this 
+library
is available.


  MARCLANG/ParallelUserAgent-2.57.tar.gz
  /usr/bin/make -- OK
Running make test
/usr/bin/perl t/TEST 
local/compatibility....Can't locate object method "parse_head" via pac
+kage "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel/User
+Agent.pm line 1497, <DAEMON> line 1.
Missing base argument at local/compatibility.t line 57
local/compatibility.... Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 20/20 subtests 
local/file.............1/4 Can't locate object method "parse_head" via
+ package "LWP::Parallel::Protocol::file" at ../blib/lib/LWP/Parallel/
+UserAgent.pm line 1497.
local/file............. Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 3/4 subtests 
local/http.............1/15 Can't locate object method "parse_head" vi
+a package "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel
+/UserAgent.pm line 1497, <DAEMON> line 1.
Missing base argument at local/http.t line 85
local/http............. Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 14/15 subtests 
local/timeouts.........1/25 Can't locate object method "parse_head" vi
+a package "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel
+/UserAgent.pm line 1497, <DAEMON> line 1.
local/timeouts......... Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 24/25 subtests 

Test Summary Report
-------------------
local/compatibility (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 20 tests but ran 0.
local/file         (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 4 tests but ran 1.
local/http         (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 15 tests but ran 1.
local/timeouts     (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 25 tests but ran 1.
Files=4, Tests=3, 31 wallclock secs ( 0.01 usr  0.05 sys +  0.29 cusr 
+ 0.73 csys =  1.08 CPU)
Result: FAIL
Failed 4/4 test programs. 0/3 subtests failed.
make: *** [test] Error 255
  MARCLANG/ParallelUserAgent-2.57.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, t
+ry:
  reports MARCLANG/ParallelUserAgent-2.57.tar.gz
Running make install
  make test had returned bad status, won't install without force
[tbrannon@devel FindALocal]$

[ikegami]

Indeed. Something I wrote at work:

[One of our major projects] uses LWP::Parallel::UserAgent. LWP::Parallel::UserAgent doesn't doesn't work with libwww-perl-5.819 (Oct 19, 2008) and newer.

LWP::Parallel::UserAgent appears to work with libwww-perl versions up libwww-perl-5.814 (Jul 24, 2008), but the installer warns that it only has been tested with libwww-perl-5.760 (older than Jun 15, 2004).

Considering LWP::Parallel::UserAgent hasn't been modified since 2003, we should be wary of our continued ability to use this module.

The fix is not obvious.

Turns out we were already moving away from it. I don't know to what.

[Anonymous Monk]

The solution is here:
http://rt.cpan.org/Public/Bug/Display.html?id=40261

Quote:
I think the module can be fixed very simply by commenting out two lines.

# $protocol->parse_head($parse_head);
# $protocol->max_size($max_size);

These are lines 1496 and 1497 of LWP/Parallel/UserAgent.pm.

[simul]

One solution is to use WWW::Curl::Simple. Here's everything I needed to convert:

#my $ua = LWP::Parallel::UserAgent->new;
package CurlToes;
use base 'WWW::Curl::Simple';
sub timeout {return 20;}
sub fatal {return 0;}
package main;
my $ua = CurlToes->new();

Unfortunately, it's a bear of an install, requiring arcane things like "Moose". I never like having to install 24 modules on a machine, just to post to a few dozen urls in parallel... but it works... and it's the closest you'll get to a drop-in replacement for LWP::Parallel;

[sri]

Mojo::Client might be helpful.

use Mojo::Client;
use Mojo::Transaction;

my $client = Mojo::Client->new;

my $tx  = Mojo::Transaction->new_get('http://labs.kraih.com');
my $tx2 = Mojo::Transaction->new_get('http://mojolicious.org');
$tx2->req->headers->expect('100-continue');
$tx2->req->body('foo bar baz');

$client->process_all($tx, $tx2);

print $tx->res->code;
print $tx2->res->code;
print $tx2->res->content->file->slurp;

AnyEvent::Mojo::Client is in the works too but not yet ready for prime time i think.

[dk]

Possibly IO::Lambda would be able to help you. One needs to invest in learning the syntax though.

use strict;
use HTTP::Request;
use IO::Lambda qw(:lambda);
use IO::Lambda::HTTP qw(http_request);

lambda {
    for my $url (
        'http://www.perl.com',
        'http://www.google.com',
        'http://www.yahoo.com',
    ) {
        context(
            HTTP::Request-> new(GET => $url),
            # list of possible options:
            #
            # async_dns    => 1,
            # auth         => 'Basic', username => 'A', password => 'B
+',
            # conn_cache   => shared LWP::ConnCache instance
            # keep_alive   => 1,
            # proxy        => [ '127.0.0.1', 8080 ],
            # timeout      => 60,
            # max_redirect => 100,
        );
        http_request {
            my $res = shift;
            if ( ref($res) and ref($res) eq 'HTTP::Response') {
                print "$url: ", length($res-> content), " bytes\n";
            } else {
                print "$url: error :$res\n";
            }
        }
    }
}-> wait;

[dokkeldepper]

Parallel::Forkmanager may be helpful.

[dexterbt1]

POE::Component::Client::HTTP and the whole POE family IMHO is worth learning. We're currently using it in production to POST/GET with a single process handling over 200+ connections.

[hanami279]

I have a project that need process HTTP request in parallel. I used LWP::Parallel::UserAgent for this purpose.
Unfortunately, this package did not work as I desired. So I need to move to other module that can handle it well. I would like someone provide me some code sample with POE module, since I have a little knowledge about POE.
Thanks