Stream file from one HTTP server to another with HTTP GET and PUT requests

[Anonymous Monk]

I do not have a lot of experience w/ perl IO or LWP, WWW::Curl and POE::Component::Client::HTTP but I am looking for a way to stream a file from one server to another.

I can acomplish it with curl but I will be copying millions of files and I am trying to avoid the overhead of all the extra forks. Plus I will be threading the whole thing using a Boss/Worker model.

Basically I'm trying to duplicate the following all in perl.

curl http://server1/path/file.dat | curl --upload-file - http://server2/upload/file.dat

The trick is the streaming part. I can get LWP and WWW::Curl::Easy to GET/PUT files from disk, but I really need to stream them. I will be moving some large files and can't afford to GET then PUT a file. Plus the disk IO may reduce my overall throughput. Holding the content in memory is also out. I really do need to stream... That's where I'm stuck.

I can probably patch something together w/ WWW::Curl::Multi but as I was looking around there was some suggestion to use POE::Component::Client::HTTP instead of WWW::Curl::Multi for someone else's task.

Regards,
-Alan

[thewebsi]

LWP's request() method accepts a callback option that does exactly this. You would do a GET and a PUT both with callbacks that read and write the data in chunks.

A GET example is provided in the LWP Cookbook, and PUT examples can be found on PerlMonks here and here.

This method does require a fork() (like the command-line counterpart). To do it without one, use Net::HTTP for the PUT.

[Anonymous Monk]

I was going to suggest using AnyEvent::HTTP but a streaming request/PUT/POST/upload isn't supported ... I think it needs to register a on_body easily

Here is my clumsy attempt which kinda seems to work , two requests are made, but I've not verified that the body/content is sent to the 2nd one -- and it hangs

#!/usr/bin/perl --
use strict;
use warnings;
use AnyEvent::HTTP;
use AnyEvent;
AnyEvent->idle( cb => sub { print "\nidling @_\n" } );
my $cv = AnyEvent->condvar(
    cb => sub {
        warn "done";
    }
);

http_get    ## schedule/cue up an event
  "http://localhost/test2",
  want_body_handle => 1,
  sub {
    warn "get one @_\n";
$cv->begin;
#~ my ($handle, $hdr) = @_;
    my( $readFrom, $hdr ) = @_;
    $readFrom->on_eof( sub { $readFrom->destroy } );
    my %headers = (
        cookie => $hdr->{'set-cookie'},       ## for my server since s
+ame
        length => $hdr->{"content-length"},
        type   => $hdr->{"content-type"},
    );

    my $just_this_once = 0;
    http_post "http://localhost/test2?whatchyagot",
      undef, # NO BODY
      headers          => \%headers,
      want_body_handle => 1,
      sub {
        return if $just_this_once;
        $just_this_once++;
        $cv->begin;
        
        warn "what is this @_\n";
        my( $writeTo, $hdr ) = @_;

        $readFrom->on_read(
            sub {
                my $data = delete $_[0]{rbuf};
                $writeTo->push_write( $data );
                return;
            }
        );
        return;
      };
    return;
  };

$cv->end;
## MainLoop/run the program (do the get_ing and post_ing )
print '$cv->recv ', $cv->recv, "\n";

__END__
get one AnyEvent::Handle=HASH(0xd0ef24) HASH(0xbcc254)
what is this AnyEvent::Handle=HASH(0xc0058c) HASH(0x9ef42c)
Terminating on signal SIGINT(2)

[Anonymous Monk]

NOPE, didn't work, body was empty, oh well