Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

LWP file transfer running out ot memory on a whopping big file

by adamsj (Hermit)
on Feb 07, 2002 at 13:44 UTC ( #143869=perlquestion: print w/ replies, xml ) Need Help??
adamsj has asked for the wisdom of the Perl Monks concerning the following question:

use strict; use LWP; my $url = 'ftp://path/to/a/greatbigfile'; my $agent = LWP::UserAgent->new; my $request = HTTP::Request->new(GET => $url); $agent->proxy('ftp' => 'http://our.proxy.server:8080'); my $response = $agent->request($request); $response->is_success or die "$url: ", $response->message,"\n"; open(MYOUT, ">/home/greatbigfile.") or die "No open? $!"; print MYOUT $response->content; close(MYOUT);

Nothing wrong with the code per se--it works fine on itty bitty files--but it fails like this:

ftp://path/to/a/greatbigfile: Out of memory during "large" request for + 33558528 bytes, total sbrk() is 37230796 bytes

on a file of nearly 40 meg. Is there any way to pull the file in chunks? I don't find it in the documentation.

adamsj

They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Comment on LWP file transfer running out ot memory on a whopping big file
Select or Download Code
Replies are listed 'Best First'.
Re: LWP file transfer running out ot memory on a whopping big file
by gav^ (Curate) on Feb 07, 2002 at 13:56 UTC
    If you type perldoc lwpcook or go here you will find loads LWP examples including one about handling large documents.

    A quick copy paste is:

    use LWP::UserAgent; $ua = LWP::UserAgent->new; my $req = HTTP::Request->new(GET => 'http://www.linpro.no/lwp/libwww-perl-5.46.tar.gz'); $res = $ua->request($req, "libwww-perl.tar.gz"); if ($res->is_success) { print "ok\n"; } else { print $res->status_line, "\n"; }

    gav^

      It's workin' like a charm--thanks!

      for ($i = 0; $i < 1000; $++) { print "I _will_ remember to read lwpcook as part of the lwp docum +entation.\n"; }

      adamsj

      They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

        for ($i = 0; $i < 1000; $++)

        Whoopsie, you forgot an 'i'.

        print "I wll remember my i's\n" for ( 0..999);
        Couldn't resist ;)

        blyman
        setenv EXINIT 'set noai ts=2'

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://143869]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2015-07-08 03:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (94 votes), past polls