sending a large file via http

[mifflin]

I have client that wants me to send him a file via http.
The potential problem here is this file may be rather large (megabytes).
The usual way that I send data through http is via the POST method like...

   my $headers = HTTP::Headers->new();
   $headers->header('Content-Type' => 'text/plain; charset=utf-8');
   $headers->header('Content-Length' => length($sendthis));
   my $request = HTTP::Request->new('POST', $url, $headers, $sendthis)
+;
   $request->protocol('HTTP/1.1');
   my $browser = LWP::UserAgent->new();
   my $response = $browser->request($request);
   my $gotthis = $response->content();

This works but can result it memory issues if the variable $sendthis is large.
Is there a better way to send large files via http?

[rhesa]

LWP::UserAgent allows you to pass a callback to the request() method, which you can use to stream data into the request.

This is the relevant quote:

$ua->request( $request, $content_cb )

You are allowed to use a CODE reference as content in the request object passed in. The content function should return the content when called. The content can be returned in chunks. The content function will be invoked repeatedly until it return an empty string to signal that there is no more content.

[mifflin]

So does this mean that the content-length header value is no longer used?
Here is some untested sample code based on your response and reading the lwp::useragent docs...

use strict;
use warnings;
use LWP;
use IO::Handle;
use Fatal qw( open close );

my $url = 'https://........';
my filename = 'file2send.txt';
my $LINES_PER_CHUNK = 100;
open my $fh, '<', $filename;

sub sendthis {

    my @buf;
    my $linesread = 0;

    if ($fh->isopened) {
        while (my $line = <$fh>) {
            $linesread++;
            push(@buf, $line);
            if ($linesread == $LINES_PER_CHUNK) {
                return join('', @buf);
            }
        }
        close $fh;
        return join('', @buf);
    }

    return '';    
}

eval {
    my $headers = HTTP::Headers->new();
    $headers->header('Content-Type' => 'text/plain; charset=utf-8');
    $headers->header('Content-Length' => -s $file2send);
    my $request = HTTP::Request->new('POST', $url, $headers);
    $request->protocol('HTTP/1.1');
    my $browser = LWP::UserAgent->new();
    my $response = $browser->request($request, \&sendthis);
    my $gotthis = $response->content();
    print $gotthis;
};
if ($@) {
    # handle error !
}

Edit: made changes that rhesa suggested regarding request method callback and content length

[rhesa]

So does this mean that the content-length header value is no longer used?

I can't tell from the LWP docs if it handles that for you, but you can still set it yourself, for example with my $length = -s $filename; and then adding the appropriate header.

Regarding your code, I read the docs slightly differently:

    my $request = HTTP::Request->new('POST', $url, $headers);
    $request->protocol('HTTP/1.1');
    my $browser = LWP::UserAgent->new();
    my $response = $browser->request($request, \&sendthis);

That is, you pass the callback to the LWP->request method, not the HTTP::Request constructor.

[Joost]

A quick scan of the docs for LWP and HTTP::Message didn't turn up anything useful. You may need some other tool to stream large files. wget has a --post-file option that looks like it might work better, but I haven't tried it myself.

[Util]

Similar to `wget`, cURL is an efficient C executable with options for POST upload via both content types application/x-www-form-urlencoded and multipart/form-data, and also supports HTTP compression. You can build a command line for system(), or use the libcurl's Perl interface module WWW::Curl. Available for Linux and Win32, and pre-installed on OS X. I use cURL every day!