Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Handling lots of PUT data in CGI

by blue_cowdawg (Monsignor)
on May 28, 2013 at 15:19 UTC ( #1035635=note: print w/ replies, xml ) Need Help??


in reply to Handling lots of PUT data in CGI

Dear Monk,
Welcome to the Monastery, I see you joined us today. Before I go further I'd like to direct your attention to some information that may be of service to you:

  • this section of our Tutorials for starters with this page in particular so you can learn your way around the Monastery.
  • Next would be this tutorial since there is a right way and many wrong ways to ask questions.
  • Specific to your question here is a good place to start

      I'm working on a REST api proxy, and I've run into a problem handling the large quantities of PUT data that come through occasionally.

The college professor in me immediately wants to know what have you tried and what is failing? Your question is awfully generic so giving you advice on how to proceed is nearly impossible. What format is your data message to your CGI script in given you state that it is not originating from an HTML form?

We're here to help, but the better you define your problem and show us what is failing will determine the quality of your experience here.


Peter L. Berghold -- Unix Professional
Peter -at- Berghold -dot- Net; AOL IM redcowdawg Yahoo IM: blue_cowdawg


Comment on Re: Handling lots of PUT data in CGI
Re^2: Handling lots of PUT data in CGI
by rufusisnodufus (Initiate) on May 28, 2013 at 19:36 UTC

    Thanks for the welcome and the suggestions as to formulate my question better.

    The problem that I'm having seems to be generic, which is why my question was phrased that way. To get into more specifics, an example I'm having issues with would be a PUT request which a huge binary file as the payload. It's a request issued with something like curl, so it's just the PUT, some headers, and then the file itself. The application type in this case would be application/octet-stream, so the data is dumped into the PUTDATA parameter by CGI.pm. The problem seems to be there aren't always enough resources available for Perl to read this into memory. Instead of dealing with that whole chunk of data as a scalar, I would want CGI to save it to disc somewhere and just point me to the file.

    Here's a super simple example of what I'm talking about:

    my $incoming_request = new CGI; print $incoming_request->param("PUTDATA");

    The above will cause Perl to run out of memory if the PUT data is large enough.

    Thanks again for your help

          The problem seems to be there aren't always enough resources available for Perl to read this into memory. Instead of dealing with that whole chunk of data as a scalar, I would want CGI to save it to disc somewhere and just point me to the file.

      To me that is a different problem than a Perl issue. Almost sounds systemic to me. If, for example, you are running this code on a system with 4Gb of RAM (pretty common these days) and you are running out of memory then you need to constrain the data to be in smaller chunks.


      Peter L. Berghold -- Unix Professional
      Peter -at- Berghold -dot- Net; AOL IM redcowdawg Yahoo IM: blue_cowdawg

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1035635]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (5)
As of 2014-09-23 01:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (210 votes), past polls