Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

LWP file transfer running out ot memory on a whopping big file

by adamsj (Hermit)
on Feb 07, 2002 at 13:44 UTC ( #143869=perlquestion: print w/ replies, xml ) Need Help??
adamsj has asked for the wisdom of the Perl Monks concerning the following question:

use strict; use LWP; my $url = 'ftp://path/to/a/greatbigfile'; my $agent = LWP::UserAgent->new; my $request = HTTP::Request->new(GET => $url); $agent->proxy('ftp' => 'http://our.proxy.server:8080'); my $response = $agent->request($request); $response->is_success or die "$url: ", $response->message,"\n"; open(MYOUT, ">/home/greatbigfile.") or die "No open? $!"; print MYOUT $response->content; close(MYOUT);

Nothing wrong with the code per se--it works fine on itty bitty files--but it fails like this:

ftp://path/to/a/greatbigfile: Out of memory during "large" request for + 33558528 bytes, total sbrk() is 37230796 bytes

on a file of nearly 40 meg. Is there any way to pull the file in chunks? I don't find it in the documentation.

adamsj

They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Comment on LWP file transfer running out ot memory on a whopping big file
Select or Download Code
Re: LWP file transfer running out ot memory on a whopping big file
by gav^ (Curate) on Feb 07, 2002 at 13:56 UTC
    If you type perldoc lwpcook or go here you will find loads LWP examples including one about handling large documents.

    A quick copy paste is:

    use LWP::UserAgent; $ua = LWP::UserAgent->new; my $req = HTTP::Request->new(GET => 'http://www.linpro.no/lwp/libwww-perl-5.46.tar.gz'); $res = $ua->request($req, "libwww-perl.tar.gz"); if ($res->is_success) { print "ok\n"; } else { print $res->status_line, "\n"; }

    gav^

      It's workin' like a charm--thanks!

      for ($i = 0; $i < 1000; $++) { print "I _will_ remember to read lwpcook as part of the lwp docum +entation.\n"; }

      adamsj

      They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

        for ($i = 0; $i < 1000; $++)

        Whoopsie, you forgot an 'i'.

        print "I wll remember my i's\n" for ( 0..999);
        Couldn't resist ;)

        blyman
        setenv EXINIT 'set noai ts=2'

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://143869]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2014-10-25 07:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (142 votes), past polls