Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

sending a large file via http

by mifflin (Curate)
on Jun 18, 2007 at 21:29 UTC ( #621871=perlquestion: print w/ replies, xml ) Need Help??
mifflin has asked for the wisdom of the Perl Monks concerning the following question:

I have client that wants me to send him a file via http.
The potential problem here is this file may be rather large (megabytes).
The usual way that I send data through http is via the POST method like...
my $headers = HTTP::Headers->new(); $headers->header('Content-Type' => 'text/plain; charset=utf-8'); $headers->header('Content-Length' => length($sendthis)); my $request = HTTP::Request->new('POST', $url, $headers, $sendthis) +; $request->protocol('HTTP/1.1'); my $browser = LWP::UserAgent->new(); my $response = $browser->request($request); my $gotthis = $response->content();
This works but can result it memory issues if the variable $sendthis is large.
Is there a better way to send large files via http?

Comment on sending a large file via http
Download Code
Re: sending a large file via http
by Joost (Canon) on Jun 18, 2007 at 21:41 UTC
      Similar to `wget`, cURL is an efficient C executable with options for POST upload via both content types application/x-www-form-urlencoded and multipart/form-data, and also supports HTTP compression. You can build a command line for system(), or use the libcurl's Perl interface module WWW::Curl. Available for Linux and Win32, and pre-installed on OS X. I use cURL every day!
Re: sending a large file via http
by rhesa (Vicar) on Jun 18, 2007 at 22:17 UTC
    LWP::UserAgent allows you to pass a callback to the request() method, which you can use to stream data into the request.

    This is the relevant quote:

    $ua->request( $request, $content_cb )

    You are allowed to use a CODE reference as content in the request object passed in. The content function should return the content when called. The content can be returned in chunks. The content function will be invoked repeatedly until it return an empty string to signal that there is no more content.
      So does this mean that the content-length header value is no longer used?
      Here is some untested sample code based on your response and reading the lwp::useragent docs...
      use strict; use warnings; use LWP; use IO::Handle; use Fatal qw( open close ); my $url = 'https://........'; my filename = 'file2send.txt'; my $LINES_PER_CHUNK = 100; open my $fh, '<', $filename; sub sendthis { my @buf; my $linesread = 0; if ($fh->isopened) { while (my $line = <$fh>) { $linesread++; push(@buf, $line); if ($linesread == $LINES_PER_CHUNK) { return join('', @buf); } } close $fh; return join('', @buf); } return ''; } eval { my $headers = HTTP::Headers->new(); $headers->header('Content-Type' => 'text/plain; charset=utf-8'); $headers->header('Content-Length' => -s $file2send); my $request = HTTP::Request->new('POST', $url, $headers); $request->protocol('HTTP/1.1'); my $browser = LWP::UserAgent->new(); my $response = $browser->request($request, \&sendthis); my $gotthis = $response->content(); print $gotthis; }; if ($@) { # handle error ! }
      Edit: made changes that rhesa suggested regarding request method callback and content length
        So does this mean that the content-length header value is no longer used?
        I can't tell from the LWP docs if it handles that for you, but you can still set it yourself, for example with my $length = -s $filename; and then adding the appropriate header.

        Regarding your code, I read the docs slightly differently:

        my $request = HTTP::Request->new('POST', $url, $headers); $request->protocol('HTTP/1.1'); my $browser = LWP::UserAgent->new(); my $response = $browser->request($request, \&sendthis);
        That is, you pass the callback to the LWP->request method, not the HTTP::Request constructor.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://621871]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2014-11-28 21:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (200 votes), past polls