Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Downloading a lot of files

by ghenry (Vicar)
on Jun 23, 2005 at 23:25 UTC ( #469561=snippet: print w/ replies, xml ) Need Help??

Description: Have a list of RPMS or Debs? Stick them in a file, one on each line, and run this. I use it when upgrading between Fedora distros, if any RPMS have been missed according to upgrade.log
#!/usr/bin/perl -w
#
# Read in upgrade.log and get missing RPMS 
#
# GH - 22.06.05
#
use strict;

use LWP::Simple;

open(RPMS, "<", "./upgrade.log")
    or die "File not found or unreadable: $!\n";

my $url = 'http://download.fedora.redhat.com/pub/fedora/linux/core/4/i
+386/os/Fedora/RPMS/';

while (<RPMS>) {
    chomp;
    print -e $_ ? "Already downloaded $_\n" : "Downloading $_\n";
    next if (-e $_);
    
    getstore ($url . $_, $_);
  
    print -e $_ ? "Got $_\n" : "Download failed for $_\n";

}
close(RPMS);

print "Download complete.\n";
Comment on Downloading a lot of files
Download Code
Re: Downloading a lot of files
by ihb (Deacon) on Jun 23, 2005 at 23:47 UTC

    &mirror could be handy here.

    use strict; use LWP::Simple; use File::Slurp qw/ read_file /; my $upgrade_file = './upgrade.log'; my $url = 'http://download.fedora.redhat.com/pub/fedora/linux/core/4/i +386/os/Fedora/RPMS/'; for (read_file($upgrade_file)) { chomp; print "Checking $_... "; my $status = mirror($url . $_, $_); print $status == RC_OK ? 'downloaded' : $status == RC_NOT_MODIFIED ? 'already downloaded' : "error: $status"; print "\n"; } print "Download complete.\n";

    ihb

    See perltoc if you don't know which perldoc to read!

Re: Downloading a lot of files
by davidrw (Prior) on Jun 24, 2005 at 01:40 UTC
    (since TWTOWTDI can include shell utils ;) ) bash syntax (though less error handling, which could partially be fixed with a test of the filename and of $?; could also tweak wget's cmdline options, or use curl instead of wget, etc):
    url='http://download.fedora.redhat.com/pub/fedora/linux/core/4/i386/os +/Fedora/RPMS/' for f in `cat upgrade.log`; do wget $url$f; done
    But obviously LWP::Simple is cooler. :)

      That way you lose the feature for persistent connections of wget. I use wget -i upgrade.log if I have a file or open WGET, "| wget -i -" if I don't know the list of URLs beforehand.

Re: Downloading a lot of files
by fmerges (Chaplain) on Jun 24, 2005 at 12:37 UTC
    Hi,

    I know this is about Perl...but, why you don use tools like rsync (http://www.rsync.org) or apt-get.

    Apt-get is wonderfull tool, there're also other tools like apt-get.

    I use rsync for so many tasks, like file syncs, mirroring, etc

    Another really interesting tool is Unison (http://www.cis.upenn.edu/~bcpierce/unison/), for syncronise host in a two directions way.

    Hope it can be useful. Regards

    :)

Back to Snippets Section

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: snippet [id://469561]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (6)
As of 2014-07-28 05:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (186 votes), past polls