Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

POST'ing a large File with LWP::UserAgent

by overrider (Acolyte)
on May 03, 2009 at 05:19 UTC ( #761560=perlquestion: print w/replies, xml ) Need Help??
overrider has asked for the wisdom of the Perl Monks concerning the following question:

Hello there, I have a Program that shall upload Files. It actually works very well with smaller Movies, but when tried for example on a 300MB File it will just crash with an Error like
Out of memory during "large" request for 536875008 bytes, total sbrk() + is 7065600 bytes at /usr/local/lib/perl5/site_perl/5.8.9/HTTP/Reques +t/ line 160.
What id like to do now is to build a callback function, so that when I POST the file, it will basically read the File chunk by chunk. That way even if i upload a 1GB Movie, Memory used stays pretty much the same. I tried to implement it like this
my $filename = "largefile.avi"; open(FH,"<$filename"); my $upload_cb = sub { if ( sysread FH, my $buffer, 4096 ){ return $buffer; } else { close FH; return ''; } } # later in the Code, when uploading the File, calling it like this $vid_fields{field_uploadfile} = [ $upload_cb ] ; $vid_fields{field_private_share_entities} = ""; $resp = upload_step_two($action_url, \%vid_fields);
When running the Program, i get a
Can't open file CODE(0x821d6d4): No such file or directory at /usr/loc +al/lib/perl5/site_perl/5.8.9/LWP/ line 401
Can someone give me a simple Example of POSTing a large File when using LWP::Useragent? Thanks a lot

Replies are listed 'Best First'.
Re: POST'ing a large File with LWP::UserAgent
by Anonymous Monk on May 03, 2009 at 06:21 UTC
    That is not a complete program.
      #!/usr/bin/perl -- use strict; use warnings; use LWP 5.825; my $url = "http://foo/upload"; my $file = __FILE__;#"foo/bar"; my $ua = LWP::UserAgent->new; $HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1; use HTTP::Request::Common; my $req = POST( $url, Content_Type => 'multipart/form-data', Content => [ file => [$file] ], ); # set up callback { my $gen = $req->content(); die unless ref($gen) eq "CODE"; my $i = 0; $req->content( sub { my $chunk = &$gen(); # get chunk of data warn $i++; return $chunk; # send it to $url } ); #use Data::Dumper;die print Data::Dumper->new([($ua,$req, $gen)])->Ind +ent(1)->Deparse(1)->Dump; } my $res = $ua->request($req); #do it print $res->status_line;

        Thanks for the feed back. Hi can you explain what the below code actually do

        # set up callback { my $gen = $req->content(); die unless ref($gen) eq "CODE"; my $i = 0; $req->content( sub { my $chunk = &$gen(); # get chunk of data warn $i++; return $chunk; # send it to $url } );

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://761560]
Approved by McDarren
Corion idly wonders about creating a series of HTTP requests. There seems to be no framework to generate a series of HTTP requests, like "try all these requests"
[Corion]: This is vaguely inspired by that list of "nasty" strings, which I'd like to replace among HTTP parameters, just to see whether the application crashes, but also for easy downloading of a list of URLs etc
[Corion]: Maybe I'll just conoct something using Algorithm::Permute to create an (OO) generator for such requests, or a simple function.
[Corion]: For testing I imagine one would want to test a random sampling of such "bad"/"unexpected" requests, while for downloading, one would want to generate them all in order, but not necessarily as a huge list

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (8)
As of 2017-01-16 15:40 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (151 votes). Check out past polls.