Re: Concurrent LWP usage (aka: LWP::PUA?)

[metaperl]

well, more news. I just tried to install LWP::Parallel::UserAgent and see that it is not updated for the latest LWP and the test suite fails...

[tbrannon@devel FindALocal]$ sudo cpan LWP::Parallel::UserAgent
)

  CPAN.pm: Going to build M/MA/MARCLANG/ParallelUserAgent-2.57.tar.gz


           **** WARNING: Unsupported Version of libwww found! ****

This library only supports libwww up to version 5.76. You seem to have
version 5.821 installed, which has not yet been tested by the package 
+author
for compatibility. Some or all tests might fail!

Install at your own risk, or wait until a more recent version of this 
+library
is available.


  MARCLANG/ParallelUserAgent-2.57.tar.gz
  /usr/bin/make -- OK
Running make test
/usr/bin/perl t/TEST 
local/compatibility....Can't locate object method "parse_head" via pac
+kage "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel/User
+Agent.pm line 1497, <DAEMON> line 1.
Missing base argument at local/compatibility.t line 57
local/compatibility.... Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 20/20 subtests 
local/file.............1/4 Can't locate object method "parse_head" via
+ package "LWP::Parallel::Protocol::file" at ../blib/lib/LWP/Parallel/
+UserAgent.pm line 1497.
local/file............. Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 3/4 subtests 
local/http.............1/15 Can't locate object method "parse_head" vi
+a package "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel
+/UserAgent.pm line 1497, <DAEMON> line 1.
Missing base argument at local/http.t line 85
local/http............. Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 14/15 subtests 
local/timeouts.........1/25 Can't locate object method "parse_head" vi
+a package "LWP::Parallel::Protocol::http" at ../blib/lib/LWP/Parallel
+/UserAgent.pm line 1497, <DAEMON> line 1.
local/timeouts......... Dubious, test returned 255 (wstat 65280, 0xff0
+0)
 Failed 24/25 subtests 

Test Summary Report
-------------------
local/compatibility (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 20 tests but ran 0.
local/file         (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 4 tests but ran 1.
local/http         (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 15 tests but ran 1.
local/timeouts     (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 25 tests but ran 1.
Files=4, Tests=3, 31 wallclock secs ( 0.01 usr  0.05 sys +  0.29 cusr 
+ 0.73 csys =  1.08 CPU)
Result: FAIL
Failed 4/4 test programs. 0/3 subtests failed.
make: *** [test] Error 255
  MARCLANG/ParallelUserAgent-2.57.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, t
+ry:
  reports MARCLANG/ParallelUserAgent-2.57.tar.gz
Running make install
  make test had returned bad status, won't install without force
[tbrannon@devel FindALocal]$

[ikegami]

Indeed. Something I wrote at work:

[One of our major projects] uses LWP::Parallel::UserAgent. LWP::Parallel::UserAgent doesn't doesn't work with libwww-perl-5.819 (Oct 19, 2008) and newer.

LWP::Parallel::UserAgent appears to work with libwww-perl versions up libwww-perl-5.814 (Jul 24, 2008), but the installer warns that it only has been tested with libwww-perl-5.760 (older than Jun 15, 2004).

Considering LWP::Parallel::UserAgent hasn't been modified since 2003, we should be wary of our continued ability to use this module.

The fix is not obvious.

Turns out we were already moving away from it. I don't know to what.

[Anonymous Monk]

The solution is here:
http://rt.cpan.org/Public/Bug/Display.html?id=40261

Quote:
I think the module can be fixed very simply by commenting out two lines.

# $protocol->parse_head($parse_head);
# $protocol->max_size($max_size);

These are lines 1496 and 1497 of LWP/Parallel/UserAgent.pm.

[simul]

One solution is to use WWW::Curl::Simple. Here's everything I needed to convert:

#my $ua = LWP::Parallel::UserAgent->new;
package CurlToes;
use base 'WWW::Curl::Simple';
sub timeout {return 20;}
sub fatal {return 0;}
package main;
my $ua = CurlToes->new();

Unfortunately, it's a bear of an install, requiring arcane things like "Moose". I never like having to install 24 modules on a machine, just to post to a few dozen urls in parallel... but it works... and it's the closest you'll get to a drop-in replacement for LWP::Parallel;