Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Stream file from one HTTP server to another with HTTP GET and PUT requests

by Anonymous Monk
on Nov 16, 2012 at 00:44 UTC ( #1004091=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I do not have a lot of experience w/ perl IO or LWP, WWW::Curl and POE::Component::Client::HTTP but I am looking for a way to stream a file from one server to another.

I can acomplish it with curl but I will be copying millions of files and I am trying to avoid the overhead of all the extra forks. Plus I will be threading the whole thing using a Boss/Worker model.

Basically I'm trying to duplicate the following all in perl.

curl http://server1/path/file.dat | curl --upload-file - http://server2/upload/file.dat

The trick is the streaming part. I can get LWP and WWW::Curl::Easy to GET/PUT files from disk, but I really need to stream them. I will be moving some large files and can't afford to GET then PUT a file. Plus the disk IO may reduce my overall throughput. Holding the content in memory is also out. I really do need to stream... That's where I'm stuck.

I can probably patch something together w/ WWW::Curl::Multi but as I was looking around there was some suggestion to use POE::Component::Client::HTTP instead of WWW::Curl::Multi for someone else's task.


Comment on Stream file from one HTTP server to another with HTTP GET and PUT requests
Download Code
Re: Stream file from one HTTP server to another with HTTP GET and PUT requests
by thewebsi (Beadle) on Nov 16, 2012 at 06:00 UTC

    LWP's request() method accepts a callback option that does exactly this. You would do a GET and a PUT both with callbacks that read and write the data in chunks.

    A GET example is provided in the LWP Cookbook, and PUT examples can be found on PerlMonks here and here.

    This method does require a fork() (like the command-line counterpart). To do it without one, use Net::HTTP for the PUT.

Re: Stream file from one HTTP server to another with HTTP GET and PUT requests (can't use AnyEvent::HTTP)
by Anonymous Monk on Nov 16, 2012 at 11:18 UTC

    I was going to suggest using AnyEvent::HTTP but a streaming request/PUT/POST/upload isn't supported ... I think it needs to register a on_body easily

    Here is my clumsy attempt which kinda seems to work , two requests are made, but I've not verified that the body/content is sent to the 2nd one -- and it hangs

    #!/usr/bin/perl -- use strict; use warnings; use AnyEvent::HTTP; use AnyEvent; AnyEvent->idle( cb => sub { print "\nidling @_\n" } ); my $cv = AnyEvent->condvar( cb => sub { warn "done"; } ); http_get ## schedule/cue up an event "http://localhost/test2", want_body_handle => 1, sub { warn "get one @_\n"; $cv->begin; #~ my ($handle, $hdr) = @_; my( $readFrom, $hdr ) = @_; $readFrom->on_eof( sub { $readFrom->destroy } ); my %headers = ( cookie => $hdr->{'set-cookie'}, ## for my server since s +ame length => $hdr->{"content-length"}, type => $hdr->{"content-type"}, ); my $just_this_once = 0; http_post "http://localhost/test2?whatchyagot", undef, # NO BODY headers => \%headers, want_body_handle => 1, sub { return if $just_this_once; $just_this_once++; $cv->begin; warn "what is this @_\n"; my( $writeTo, $hdr ) = @_; $readFrom->on_read( sub { my $data = delete $_[0]{rbuf}; $writeTo->push_write( $data ); return; } ); return; }; return; }; $cv->end; ## MainLoop/run the program (do the get_ing and post_ing ) print '$cv->recv ', $cv->recv, "\n"; __END__ get one AnyEvent::Handle=HASH(0xd0ef24) HASH(0xbcc254) what is this AnyEvent::Handle=HASH(0xc0058c) HASH(0x9ef42c) Terminating on signal SIGINT(2)
      NOPE, didn't work, body was empty, oh well

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1004091]
Approved by Athanasius
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (7)
As of 2014-07-24 06:36 GMT
Find Nodes?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:

    Results (158 votes), past polls