Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Sending large file contents in SOAP call using SOAP::Lite

by Anonymous Monk
on Oct 17, 2013 at 19:20 UTC ( #1058670=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am using SOAP::Lite (latest version) to make call to a webservice with large file (about 50MB). I am using following code. I see that when making soap call the memory usage is very high (approx 1.5GB). And if I go above 60MB, I am running out of memory and perl scripting is exiting with out making the SOAP call. Is this the error in the SOAP::Lite module or I am using it the wrong way. Also, what is a better way to make a SOAP call with large data. Please see below for the code I am currently using:

#Read the file my $value = read_file( $attachmentFilename , binmode => ':raw', scalar +_ref => 1 ); my @params = ( $header, SOAP::Data->name(fileName => $attachFilename), SOAP::Data->name(taskDataContent => encode_base64($$val +ue)), #base64 encode the file contents SOAP::Data->name(results => $results)); #Call the webservice with SOAP::Lite $returnResult = $soap->call($ptrMethod => @params);

Any help is greatly appreciated.

Thanks, Ana

Comment on Sending large file contents in SOAP call using SOAP::Lite
Download Code
Re: Sending large file contents in SOAP call using SOAP::Lite
by Anonymous Monk on Oct 18, 2013 at 01:36 UTC

    Is this the error in the SOAP::Lite module or I am using it the wrong way.

    Yeah, 1.5GB seems like there might be a problem ... but not where, I doubt its within SOAP::Lite

    perl -Mojo -e " b( shift )->slurp->b64_encode->say; int <>; " 49_678_352_bytesfile > 65M-b64ed

    This takes about ~130mb on my machine, and around 20mb of that is perl -Mojo itself alone

    I can't imagine SOAP::Lite tacking eleven times that amount, maybe one or three temporarily, but not eleven

    Without mojo, with File::Slurp+SOAP::Lite, to slurp this 65M b64ed file takes about 74mb, 1500mb/74mb = 20 so still nowhere in the ballpark

    I don't feel like looking into it past this; You can try

    $soap->serializer()->envelope( method => 'DoThis', $param ),
    to see how much memory it takes to compose that request you're using -- shouldn't be 1.5 GB

    I imagine it is the response that you're getting (your program is getting) to that request is an xml bomb ... or something else entirely

    Also, what is a better way to make a SOAP call with large data

    I imagine maybe https://metacpan.org/module/SOAP::Lite#ATTACHMENTS,

    or even XML::Compile::XOP

    but if your server doesn't accept that :)

    Good luck

      I tried serializer code you mentioned on a 34mb file and it almost used up 500MB while it is serializing and I printed the length of the resultant serialized text 47mb. So, the issue is not the size of the SOAP message it is sending or receiving, it is that serialization itself using up too much memory. Do you or anyone see the same issue?
Re: Sending large file contents in SOAP call using SOAP::Lite
by keszler (Priest) on Oct 18, 2013 at 01:40 UTC
    It's been a while since I last used SOAP::Lite, but my old code from then looked like
    my $soap = SOAP::Lite -> uri('http://server.name/nas/') -> outputxml(1) ; $resp = $soap->call('nas:show_session_ids'); $resp = $soap->call('nas:add_device' => SOAP::Data->name(sessionid => $sid )->prefix('nas'), SOAP::Data->name(ip => $ip )->prefix('nas'), SOAP::Data->name(comment => $comment )->prefix('nas'), SOAP::Data->name(hostname => $hostname )->prefix('nas'), SOAP::Data->name(domain => $domain )->prefix('nas'), SOAP::Data->name(location => $location )->prefix('nas'), SOAP::Data->name(unmanaged => $unmanaged)->prefix('nas'), SOAP::Data->name(accessmethods => $accessm )->prefix('nas'), );
    Depending on what is in $header, how you initialized $soap, and whether or not you need a namespace (e.g. 'nas'), your example code could be correct.

    It does not seem reasonable to me that a 50MB file could cause 1.5GB of memory usage. I'd start troubleshooting by looking into $value. Was your SOAP::Data->name(taskDataContent => encode_base64($$value)) a typo, or is it actually $$value because read_file is returning a reference vs a string?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1058670]
Approved by keszler
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (5)
As of 2014-07-30 23:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (241 votes), past polls