Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Download a file from website

by rahulruns (Sexton)
on Sep 13, 2012 at 07:38 UTC ( #993399=perlquestion: print w/ replies, xml ) Need Help??
rahulruns has asked for the wisdom of the Perl Monks concerning the following question:

Need to download a file from a website I have tried using LWP::Simple and curl but the connection closes before the entire file is downloaded. The file size is around 20MB and only around 6MB is downloaded

LWP::Simple code use LWP::Simple; my $url = 'https://cwiki.apache.org/confluence/display/PIG/PigTutoria +l'; my $file = 'pigtutorial.tar.gz'; my $status = getstore($url, $file); die "Error $status on $url" unless is_success($status); CURL code my $url = `curl -C - -o pigtutorial.tar.gz https://cwiki.apache.org/co +nfluence/display/PIG/PigTutorial`;

Comment on Download a file from website
Download Code
Re: Download a file from website
by Anonymous Monk on Sep 13, 2012 at 07:54 UTC

    If the remote server is timing you out, there is nothing you can do, esp if it doesn't support resuming

    wget -c -O filename -- http...

    sub LWP::UserAgent::resume

      I went through the code provide for LWP::UserAgent but it is taking time to make out where do I need to change the specific URL and filename. Could you help on that part

Re: Download a file from website
by frozenwithjoy (Curate) on Sep 13, 2012 at 08:06 UTC
    Does it work if you try to do it outside of Perl? When I run your curl command in terminal (curl -C - -o pigtutorial.tar.gz https://cwiki.apache.org/confluence/display/PIG/PigTutorial), I just get a 65kb file that is the html of the site (and not a tarball). Are you pointing to the correct place?

      Yes it works outside PERL. I am able to download the file and extract it

      rightly replied by frozenwith joy. When you check the file downloaded file from the link it has a href for the location of exact file and now it downloads from that location mentioned in href. This thread could be closed

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://993399]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2014-08-21 05:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (127 votes), past polls