Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Http Clients errors from LWP::Protocol::http after module upgrade

by tharple (Acolyte)
on Oct 18, 2012 at 21:44 UTC ( #999824=perlquestion: print w/replies, xml ) Need Help??
tharple has asked for the wisdom of the Perl Monks concerning the following question:

Performance/load testing a MS Web service, where the data generator client machine has ~100 perl scripts reading data from a local socket and then writing it to the IIS 7.5 web server. The data generator is WIN 2008 R2 with 16 core 32GB server with qty 10 Gigabit adapters. The Web server is WIN 2008 R2 with 8 core 4GB server with qty 1 Gigabit adapters. The messages are sent out via https and are distributed over all adapters. I have successfully uploaded 100,000's of 2 MB messages, but was occasionally receiving the following message:

500 write failed: Content-Type: text/plain client-date: Wed, 17 Oct 20 +12 16:02:37 GMT client-warning: Internal response 500 write failed:

My investigation lead to Stacktrace/Perlmonks suggesting an upgrade of Net::SSLeay and Bundle:LWP After doing so my level of stability has decrease and I now receive significantly more of the following error:

500 Can't use an undefined value as a SCALAR reference Content-Type: text/plain client-date: Thu, 18 Oct 2012 18:15:59 GMT client-warning: Internal response Can't use an undefined value as a SCALAR reference at C:/strawberry/pe +rl/site/lib/LWP/Protocol/ line 353. Results:Can't use an undefined value as a SCALAR reference at C:/straw +berry/perl/site/lib/LWP/Protocol/ line 353.

I added a debug statement prior to the error line in c:/strawberry/perl/site/lib/LWP/Protocol/

print Dumper $wbits unless $wbits; if (defined($wbits) && $wbits =~ /[^\0]/) {

I only get the error when Dumper says $wbits is undef. It looks like $wbits should fail the defined call and not check the match condition.??? The perl scripts are separate processes started via Win32::Process::Create. Here is the code fragment making the calls:

use strict; use warnings; use IO::Socket; use LWP::UserAgent; use LWP::Protocol::http; use HTTP::Request; use Data::Dumper; use Dis::Time; use Storable qw(store_fd fd_retrieve); $ENV{HTTPS_DEBUG} = 1; .... my $userAgent = LWP::UserAgent->new(agent => '', keep_alive => 1, local_address => $localAddr, ssl_opts => { verify_hostname => 0 }, timeout => 1, ); push(@LWP::Protocol::http::EXTRA_SOCK_OPTS, SendTE => 0); my $heads = HTTP::Headers->new( Content_Type => $data->{Soap}{Content}, Accept_Encoding => $data->{Soap}{Encoding}, Expect => '100-continue', SOAPAction => $data->{Soap}{Ns}{tem}.$data->{Soap}{Method}, ); print "Executing: HTTP::Request\n" if $debug; my $req = HTTP::Request->new("POST", $data->{Soap}{Uri},$heads, $d +ata->{Message}); $data->{Retry} = 0; while(1) { $data->{Start} = Dis::Time->new(); print "Executing: HTTP::Response\n" if $debug; $data->{Response} = $userAgent->request($req); my $code = $data->{Response}->code; print "Responce code $code Recieved\n" if $debug; last if ("200" eq $code || "500" ne $code || $data->{Retry}>=$ +retry); $data->{Retry}++; print $data->{Response}->as_string,"Retrying...\n" if $debug; } ....

The retry loop and having the timeout set to 1 are workarounds added to get around the issue. This being such a strange error and in an LWP module is my motivation for the posing the question.

perl -V Summary of my perl5 (revision 5 version 12 subversion 3) configuration +: Platform: osname=MSWin32, osvers=6.1, archname=MSWin32-x64-multi-thread uname='Win32 strawberryperl #1 Sun May 15 09:43:50 2011 x +64' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=und +ef use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags =' -s -O2 -DWIN32 -DHAVE_DES_FCRYPT -DWIN64 -DCO +NSERVATIVE -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL _IMPLICIT_SYS -fno-strict-aliasing -mms-bitfields -DPERL_MSVCRT_READFI +X', optimize='-s -O2', cppflags='-DWIN32' ccversion='', gccversion='4.4.3', gccosandvers='' intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='lo +ng long', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='g++', ldflags ='-s -L"C:\strawberry\perl\lib\CORE" -L"C:\straw +berry\c\lib"' libpth=C:\strawberry\c\lib C:\strawberry\c\x86_64-w64-mingw32\lib libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 +-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws 2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdl +g32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 libc=, so=dll, useshrplib=true, libperl=libperl512.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE +" -L"C:\strawberry\c\lib"' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_64_BIT_I +NT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PE +RL_ATOF USE_SITECUSTOMIZE Built under MSWin32 Compiled at May 15 2011 09:52:35 %ENV: PERL_JSON_BACKEND="JSON::XS" PERL_YAML_BACKEND="YAML" @INC: C:/strawberry/perl/site/lib C:/strawberry/perl/vendor/lib C:/strawberry/perl/lib .

And my question are...

    defined($wbits) && $wbits =~ /^\0/) {
    In what situation can the 1st predicate pass when $wbits is undef?
    Or why would && not short circuit correctly?
    Is there something obvious that I am missing, that you see?
    If this looks like a real issue, how can I assist is further troubleshooting this?

Replies are listed 'Best First'.
Re: Http Clients errors from LWP::Protocol::http after module upgrade
by mbethke (Hermit) on Oct 19, 2012 at 01:27 UTC
    Hi tharple,

    that's a very strange error.

    In what situation can the 1st predicate pass when $wbits is undef?
    Or why would && not short circuit correctly?
    I can't think of any at all, that would make both the defined() test and && pretty pointless. What's more, line line doesn't match the error message. If something made defined() pass an undefined value, I'd have expected "Use of uninitialized value $fbits in pattern match (m//)", however the error speaks of "SCALAR reference" as if some dereferencing was going on there, which there isn't. Perl 5.14+ does some automatic dereferencing in some situations but with 5.12 that shouldn't be the case either. Frankly I don't have a clue. Perhaps something is screwing¹ with your compiled code. I'd try and split up that condition to see what exactly is acting up:
    if (defined($wbits)) { print "\$wbits is defined: ",Dumper($wbits); if($wbits =~ /[^\0]/) { print "\$wbits matched\n"; ... } else { print "\$wbits did not match\n"; } }
    ¹ To whom it may concern: non-sexual readings preferred here

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://999824]
Approved by nemesdani
Front-paged by ig
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2018-03-20 15:55 GMT
Find Nodes?
    Voting Booth?
    When I think of a mole I think of:

    Results (254 votes). Check out past polls.