Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

Parallel LWP usage

by metaperl (Curate)
on Apr 20, 2009 at 15:31 UTC ( #758739=perlquestion: print w/replies, xml ) Need Help??
metaperl has asked for the wisdom of the Perl Monks concerning the following question:

I'm interested in posting to 5 or 6 HTTP servers concurrently and getting back responses.

This is a coarse-grained parallel operation and the LWP::ParallelUserAgent example is exactly what I need.

But I notice several unresolved bugs for this package.

Further, it has not seen an update in 5 years

So, I'm curious as to how you would handle such a functional requirement... is there a more popular module for this? One with fewer bugs and more recent releases? Do I need to write my own thang in POE similar to what merlyn shows here?

Replies are listed 'Best First'.
Re: Concurrent LWP usage (aka: LWP::PUA?)
by metaperl (Curate) on Apr 20, 2009 at 16:00 UTC
    Possum in #perl-help on turned me on to AnyEvent::HTTP - it looks super duper!

      ++ I recently used this package, it's great. Highly recommend give it a try. Also WWW::Curl::Multi may be useful for you.

Re: Concurrent LWP usage (aka: LWP::PUA?)
by metaperl (Curate) on Apr 20, 2009 at 15:38 UTC
    well, more news. I just tried to install LWP::Parallel::UserAgent and see that it is not updated for the latest LWP and the test suite fails...
    [tbrannon@devel FindALocal]$ sudo cpan LWP::Parallel::UserAgent ) Going to build M/MA/MARCLANG/ParallelUserAgent-2.57.tar.gz **** WARNING: Unsupported Version of libwww found! **** This library only supports libwww up to version 5.76. You seem to have version 5.821 installed, which has not yet been tested by the package +author for compatibility. Some or all tests might fail! Install at your own risk, or wait until a more recent version of this +library is available. MARCLANG/ParallelUserAgent-2.57.tar.gz /usr/bin/make -- OK Running make test /usr/bin/perl t/TEST local/compatibility....Can't locate object method "parse_head" via pac +kage "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel/User line 1497, <DAEMON> line 1. Missing base argument at local/compatibility.t line 57 local/compatibility.... Dubious, test returned 255 (wstat 65280, 0xff0 +0) Failed 20/20 subtests local/file.............1/4 Can't locate object method "parse_head" via + package "LWP::Parallel::Protocol::file" at ../blib/lib/LWP/Parallel/ line 1497. local/file............. Dubious, test returned 255 (wstat 65280, 0xff0 +0) Failed 3/4 subtests local/http.............1/15 Can't locate object method "parse_head" vi +a package "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel +/ line 1497, <DAEMON> line 1. Missing base argument at local/http.t line 85 local/http............. Dubious, test returned 255 (wstat 65280, 0xff0 +0) Failed 14/15 subtests local/timeouts.........1/25 Can't locate object method "parse_head" vi +a package "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel +/ line 1497, <DAEMON> line 1. local/timeouts......... Dubious, test returned 255 (wstat 65280, 0xff0 +0) Failed 24/25 subtests Test Summary Report ------------------- local/compatibility (Wstat: 65280 Tests: 0 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 20 tests but ran 0. local/file (Wstat: 65280 Tests: 1 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 4 tests but ran 1. local/http (Wstat: 65280 Tests: 1 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 15 tests but ran 1. local/timeouts (Wstat: 65280 Tests: 1 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 25 tests but ran 1. Files=4, Tests=3, 31 wallclock secs ( 0.01 usr 0.05 sys + 0.29 cusr + 0.73 csys = 1.08 CPU) Result: FAIL Failed 4/4 test programs. 0/3 subtests failed. make: *** [test] Error 255 MARCLANG/ParallelUserAgent-2.57.tar.gz /usr/bin/make test -- NOT OK //hint// to see the cpan-testers results for installing this module, t +ry: reports MARCLANG/ParallelUserAgent-2.57.tar.gz Running make install make test had returned bad status, won't install without force [tbrannon@devel FindALocal]$

      Indeed. Something I wrote at work:

      [One of our major projects] uses LWP::Parallel::UserAgent. LWP::Parallel::UserAgent doesn't doesn't work with libwww-perl-5.819 (Oct 19, 2008) and newer.

      LWP::Parallel::UserAgent appears to work with libwww-perl versions up libwww-perl-5.814 (Jul 24, 2008), but the installer warns that it only has been tested with libwww-perl-5.760 (older than Jun 15, 2004).

      Considering LWP::Parallel::UserAgent hasn't been modified since 2003, we should be wary of our continued ability to use this module.

      The fix is not obvious.

      Turns out we were already moving away from it. I don't know to what.

      One solution is to use WWW::Curl::Simple. Here's everything I needed to convert:
      #my $ua = LWP::Parallel::UserAgent->new; package CurlToes; use base 'WWW::Curl::Simple'; sub timeout {return 20;} sub fatal {return 0;} package main; my $ua = CurlToes->new();
      Unfortunately, it's a bear of an install, requiring arcane things like "Moose". I never like having to install 24 modules on a machine, just to post to a few dozen urls in parallel... but it works... and it's the closest you'll get to a drop-in replacement for LWP::Parallel;
Re: Parallel LWP usage
by sri (Vicar) on Apr 21, 2009 at 03:35 UTC
    Mojo::Client might be helpful.
    use Mojo::Client; use Mojo::Transaction; my $client = Mojo::Client->new; my $tx = Mojo::Transaction->new_get(''); my $tx2 = Mojo::Transaction->new_get(''); $tx2->req->headers->expect('100-continue'); $tx2->req->body('foo bar baz'); $client->process_all($tx, $tx2); print $tx->res->code; print $tx2->res->code; print $tx2->res->content->file->slurp;
    AnyEvent::Mojo::Client is in the works too but not yet ready for prime time i think.
Re: Parallel LWP usage
by dk (Chaplain) on Apr 21, 2009 at 10:17 UTC
    Possibly IO::Lambda would be able to help you. One needs to invest in learning the syntax though.
    use strict; use HTTP::Request; use IO::Lambda qw(:lambda); use IO::Lambda::HTTP qw(http_request); lambda { for my $url ( '', '', '', ) { context( HTTP::Request-> new(GET => $url), # list of possible options: # # async_dns => 1, # auth => 'Basic', username => 'A', password => 'B +', # conn_cache => shared LWP::ConnCache instance # keep_alive => 1, # proxy => [ '', 8080 ], # timeout => 60, # max_redirect => 100, ); http_request { my $res = shift; if ( ref($res) and ref($res) eq 'HTTP::Response') { print "$url: ", length($res-> content), " bytes\n"; } else { print "$url: error :$res\n"; } } } }-> wait;
Re: Parallel LWP usage
by dokkeldepper (Friar) on Apr 20, 2009 at 17:59 UTC
Re: Parallel LWP usage
by dexterbt1 (Novice) on Apr 21, 2009 at 09:00 UTC
    POE::Component::Client::HTTP and the whole POE family IMHO is worth learning. We're currently using it in production to POST/GET with a single process handling over 200+ connections.
      I have a project that need process HTTP request in parallel. I used LWP::Parallel::UserAgent for this purpose.
      Unfortunately, this package did not work as I desired. So I need to move to other module that can handle it well. I would like someone provide me some code sample with POE module, since I have a little knowledge about POE.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://758739]
Approved by ikegami
Front-paged by Arunbear
[amitkumarj441]: I need to ask a question, where can I post my problem?
[Discipulus]: everyone is me
[Discipulus]: Seekers of Perl Wisdom is the for asking question
[Discipulus]: be sure to use code tags for code and data
[Discipulus]: and to show some effort while asking
[Discipulus]: <code> code and data</code>
[Discipulus]: amitkumarj441 more atention you put asking, more attentio you got in answers
Discipulus 's hoping he needs a oneliner ;=)

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (4)
As of 2017-04-26 20:23 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (488 votes). Check out past polls.