Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Stream file from one HTTP server to another with HTTP GET and PUT requests

by Anonymous Monk
on Nov 16, 2012 at 00:44 UTC ( #1004091=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I do not have a lot of experience w/ perl IO or LWP, WWW::Curl and POE::Component::Client::HTTP but I am looking for a way to stream a file from one server to another.

I can acomplish it with curl but I will be copying millions of files and I am trying to avoid the overhead of all the extra forks. Plus I will be threading the whole thing using a Boss/Worker model.

Basically I'm trying to duplicate the following all in perl.

curl http://server1/path/file.dat | curl --upload-file - http://server2/upload/file.dat

The trick is the streaming part. I can get LWP and WWW::Curl::Easy to GET/PUT files from disk, but I really need to stream them. I will be moving some large files and can't afford to GET then PUT a file. Plus the disk IO may reduce my overall throughput. Holding the content in memory is also out. I really do need to stream... That's where I'm stuck.

I can probably patch something together w/ WWW::Curl::Multi but as I was looking around there was some suggestion to use POE::Component::Client::HTTP instead of WWW::Curl::Multi for someone else's task.


  • Comment on Stream file from one HTTP server to another with HTTP GET and PUT requests
  • Download Code

Replies are listed 'Best First'.
Re: Stream file from one HTTP server to another with HTTP GET and PUT requests
by thewebsi (Scribe) on Nov 16, 2012 at 06:00 UTC

    LWP's request() method accepts a callback option that does exactly this. You would do a GET and a PUT both with callbacks that read and write the data in chunks.

    A GET example is provided in the LWP Cookbook, and PUT examples can be found on PerlMonks here and here.

    This method does require a fork() (like the command-line counterpart). To do it without one, use Net::HTTP for the PUT.

Re: Stream file from one HTTP server to another with HTTP GET and PUT requests (can't use AnyEvent::HTTP)
by Anonymous Monk on Nov 16, 2012 at 11:18 UTC

    I was going to suggest using AnyEvent::HTTP but a streaming request/PUT/POST/upload isn't supported ... I think it needs to register a on_body easily

    Here is my clumsy attempt which kinda seems to work , two requests are made, but I've not verified that the body/content is sent to the 2nd one -- and it hangs

    #!/usr/bin/perl -- use strict; use warnings; use AnyEvent::HTTP; use AnyEvent; AnyEvent->idle( cb => sub { print "\nidling @_\n" } ); my $cv = AnyEvent->condvar( cb => sub { warn "done"; } ); http_get ## schedule/cue up an event "http://localhost/test2", want_body_handle => 1, sub { warn "get one @_\n"; $cv->begin; #~ my ($handle, $hdr) = @_; my( $readFrom, $hdr ) = @_; $readFrom->on_eof( sub { $readFrom->destroy } ); my %headers = ( cookie => $hdr->{'set-cookie'}, ## for my server since s +ame length => $hdr->{"content-length"}, type => $hdr->{"content-type"}, ); my $just_this_once = 0; http_post "http://localhost/test2?whatchyagot", undef, # NO BODY headers => \%headers, want_body_handle => 1, sub { return if $just_this_once; $just_this_once++; $cv->begin; warn "what is this @_\n"; my( $writeTo, $hdr ) = @_; $readFrom->on_read( sub { my $data = delete $_[0]{rbuf}; $writeTo->push_write( $data ); return; } ); return; }; return; }; $cv->end; ## MainLoop/run the program (do the get_ing and post_ing ) print '$cv->recv ', $cv->recv, "\n"; __END__ get one AnyEvent::Handle=HASH(0xd0ef24) HASH(0xbcc254) what is this AnyEvent::Handle=HASH(0xc0058c) HASH(0x9ef42c) Terminating on signal SIGINT(2)
      NOPE, didn't work, body was empty, oh well

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1004091]
Approved by Athanasius
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (7)
As of 2017-09-25 19:56 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (288 votes). Check out past polls.