Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re: Mirror a package repository on the fly.

by 5mi11er (Deacon)
on May 27, 2008 at 17:26 UTC ( #688706=note: print w/ replies, xml ) Need Help??


in reply to Mirror a package repository on the fly.

I had to actually run this from the /tmp directory...

cd /tmp
perl /root/bin/automirror.pl -l http://localhost/ -e http://mirror.centos.org/ -r /opt/mirror_root/  &
And command run with logging intermixed...
[root@vr-mgmt2 centos]# yum update Loading "fastestmirror" plugin Loading "installonlyn" plugin Setting up Update Process Setting up repositories Loading mirror speeds from cached hostfile Reading repository metadata in from local files Resolving Dependencies --> Populating transaction set with selected packages. Please wait. ---> Downloading header for kpartx to pack into transaction set. [Fri May 23 10:54:50 2008] (31637) External mirror for /centos/5/updat +es/i386/RPMS/kpartx-0.4.7-12.el5_1.4.i386.rpm: kpartx-0.4.7-12.el5_1.4.i 100% |=========================| 7.5 kB 0 +0:00 ---> Package kpartx.i386 0:0.4.7-12.el5_1.4 set to be updated ---> Downloading header for libxslt to pack into transaction set. [Fri May 23 10:54:50 2008] (31638) External mirror for /centos/5/updat +es/i386/RPMS/libxslt-1.1.17-2.el5_1.1.i386.rpm: libxslt-1.1.17-2.el5_1.1. 100% |=========================| 16 kB 0 +0:00 ---> Package libxslt.i386 0:1.1.17-2.el5_1.1 set to be updated ---> Downloading header for device-mapper-multipath to pack into trans +action set. [Fri May 23 10:54:50 2008] (31625) External mirror for /centos/5/updat +es/i386/RPMS/device-mapper-multipath-0.4.7-12.el5_1.4.i386.rpm: device-mapper-multipath-0 100% |=========================| 12 kB 0 +0:00 ---> Package device-mapper-multipath.i386 0:0.4.7-12.el5_1.4 set to be + updated ---> Downloading header for kernel to pack into transaction set. [Fri May 23 10:54:50 2008] (31626) External mirror for /centos/5/updat +es/i386/RPMS/kernel-2.6.18-53.1.21.el5.i686.rpm: kernel-2.6.18-53.1.21.el5 100% |=========================| 261 kB 0 +0:00 ---> Package kernel.i686 0:2.6.18-53.1.21.el5 set to be installed ---> Downloading header for gnutls to pack into transaction set. [Fri May 23 10:54:51 2008] (31627) External mirror for /centos/5/updat +es/i386/RPMS/gnutls-1.4.1-3.el5_1.i386.rpm: gnutls-1.4.1-3.el5_1.i386 100% |=========================| 6.8 kB 0 +0:00 ---> Package gnutls.i386 0:1.4.1-3.el5_1 set to be updated ---> Downloading header for kernel-headers to pack into transaction se +t. [Fri May 23 10:54:51 2008] (31624) External mirror for /centos/5/updat +es/i386/RPMS/kernel-headers-2.6.18-53.1.21.el5.i386.rpm: kernel-headers-2.6.18-53. 100% |=========================| 175 kB 0 +0:00 ---> Package kernel-headers.i386 0:2.6.18-53.1.21.el5 set to be update +d ---> Downloading header for libvorbis to pack into transaction set. [Fri May 23 10:54:51 2008] (31631) External mirror for /centos/5/updat +es/i386/RPMS/libvorbis-1.1.2-3.el5_1.2.i386.rpm: libvorbis-1.1.2-3.el5_1.2 100% |=========================| 7.4 kB 0 +0:00 ---> Package libvorbis.i386 1:1.1.2-3.el5_1.2 set to be updated ---> Downloading header for mcstrans to pack into transaction set. [Fri May 23 10:54:51 2008] (31636) External mirror for /centos/5/updat +es/i386/RPMS/mcstrans-0.2.7-1.el5.i386.rpm: mcstrans-0.2.7-1.el5.i386 100% |=========================| 5.3 kB 0 +0:00 ---> Package mcstrans.i386 0:0.2.7-1.el5 set to be updated --> Running transaction check Dependencies Resolved ====================================================================== +======= Package Arch Version Repository + Size ====================================================================== +======= Installing: kernel i686 2.6.18-53.1.21.el5 updates + 13 M Updating: device-mapper-multipath i386 0.4.7-12.el5_1.4 updates + 2.0 M gnutls i386 1.4.1-3.el5_1 updates + 350 k kernel-headers i386 2.6.18-53.1.21.el5 updates + 789 k kpartx i386 0.4.7-12.el5_1.4 updates + 404 k libvorbis i386 1:1.1.2-3.el5_1.2 updates + 194 k libxslt i386 1.1.17-2.el5_1.1 updates + 485 k mcstrans i386 0.2.7-1.el5 updates + 16 k Transaction Summary ====================================================================== +======= Install 1 Package(s) Update 7 Package(s) Remove 0 Package(s) Total download size: 17 M Is this ok [y/N]: y Downloading Packages: [Fri May 23 10:55:03 2008] (31646) External mirror for /centos/5/updat +es/i386/RPMS/kpartx-0.4.7-12.el5_1.4.i386.rpm: (1/8): kpartx-0.4.7-12.el 100% |=========================| 404 kB 0 +0:00 [Fri May 23 10:55:03 2008] (31647) External mirror for /centos/5/updat +es/i386/RPMS/libxslt-1.1.17-2.el5_1.1.i386.rpm: (2/8): libxslt-1.1.17-2.e 100% |=========================| 485 kB 0 +0:00 [Fri May 23 10:55:04 2008] (31648) External mirror for /centos/5/updat +es/i386/RPMS/device-mapper-multipath-0.4.7-12.el5_1.4.i386.rpm: (3/8): device-mapper-mult 100% |=========================| 2.0 MB 0 +0:03 [Fri May 23 10:55:07 2008] (31649) External mirror for /centos/5/updat +es/i386/RPMS/kernel-2.6.18-53.1.21.el5.i686.rpm: (4/8): kernel-2.6.18-53.1 100% |=========================| 13 MB 0 +0:23 http://mirror.centos.org/centos/5/updates/i386/RPMS/gnutls-1.4.1-3.el5 +_1.i386.rpm: [Errno 12] Timeout: <urlopen error timed out> Trying other mirror. http://mirror.centos.org/centos/5/updates/i386/RPMS/kernel-headers-2.6 +.18-53.1.21.el5.i386.rpm: [Errno 12] Timeout: <urlopen error timed ou +t> Trying other mirror. http://mirror.centos.org/centos/5/updates/i386/RPMS/libvorbis-1.1.2-3. +el5_1.2.i386.rpm: [Errno 12] Timeout: <urlopen error timed out> Trying other mirror. http://mirror.centos.org/centos/5/updates/i386/RPMS/mcstrans-0.2.7-1.e +l5.i386.rpm: [Errno 12] Timeout: <urlopen error timed out> Trying other mirror. Error Downloading Packages: kernel-headers - 2.6.18-53.1.21.el5.i386: failure: RPMS/kernel-heade +rs-2.6.18-53.1.21.el5.i386.rpm from updates: [Errno 256] No more mirr +ors to try. mcstrans - 0.2.7-1.el5.i386: failure: RPMS/mcstrans-0.2.7-1.el5.i386 +.rpm from updates: [Errno 256] No more mirrors to try. gnutls - 1.4.1-3.el5_1.i386: failure: RPMS/gnutls-1.4.1-3.el5_1.i386 +.rpm from updates: [Errno 256] No more mirrors to try. libvorbis - 1:1.1.2-3.el5_1.2.i386: failure: RPMS/libvorbis-1.1.2-3. +el5_1.2.i386.rpm from updates: [Errno 256] No more mirrors to try. [Fri May 23 10:57:31 2008] (31637) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31646) ERROR: Getting request failed: Clie +nt closed [Fri May 23 10:57:31 2008] (31647) ERROR: Getting request failed: Clie +nt closed [Fri May 23 10:57:31 2008] (31648) ERROR: Getting request failed: Clie +nt closed [Fri May 23 10:57:31 2008] (31638) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31625) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31626) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31627) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31624) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31631) ERROR: Getting request failed: sysr +ead: Connection reset by peer [Fri May 23 10:57:31 2008] (31636) ERROR: Getting request failed: sysr +ead: Connection reset by peer [root@vr-mgmt2 centos]# [Fri May 23 10:57:31 2008] (31647) External mi +rror for /centos/5/updates/i386/RPMS/libvorbis-1.1.2-3.el5_1.2.i386.r +pm: [Fri May 23 10:57:31 2008] (31646) External mirror for /centos/5/updat +es/i386/RPMS/kernel-headers-2.6.18-53.1.21.el5.i386.rpm: [Fri May 23 10:57:31 2008] (31648) External mirror for /centos/5/updat +es/i386/RPMS/mcstrans-0.2.7-1.el5.i386.rpm: [Fri May 23 10:57:31 2008] (31637) External mirror for /centos/5/updat +es/i386/RPMS/gnutls-1.4.1-3.el5_1.i386.rpm: [Fri May 23 10:57:31 2008] (31649) ERROR: Getting request failed: Clie +nt closed yum update Loading "fastestmirror" plugin Loading "installonlyn" plugin Setting up Update Process Setting up repositories Loading mirror speeds from cached hostfile Reading repository metadata in from local files Resolving Dependencies --> Populating transaction set with selected packages. Please wait. ---> Package kpartx.i386 0:0.4.7-12.el5_1.4 set to be updated ---> Package libxslt.i386 0:1.1.17-2.el5_1.1 set to be updated ---> Package device-mapper-multipath.i386 0:0.4.7-12.el5_1.4 set to be + updated ---> Package kernel.i686 0:2.6.18-53.1.21.el5 set to be installed ---> Package gnutls.i386 0:1.4.1-3.el5_1 set to be updated ---> Package kernel-headers.i386 0:2.6.18-53.1.21.el5 set to be update +d ---> Package libvorbis.i386 1:1.1.2-3.el5_1.2 set to be updated ---> Package mcstrans.i386 0:0.2.7-1.el5 set to be updated --> Running transaction check Dependencies Resolved ====================================================================== +======= Package Arch Version Repository + Size ====================================================================== +======= Installing: kernel i686 2.6.18-53.1.21.el5 updates + 13 M Updating: device-mapper-multipath i386 0.4.7-12.el5_1.4 updates + 2.0 M gnutls i386 1.4.1-3.el5_1 updates + 350 k kernel-headers i386 2.6.18-53.1.21.el5 updates + 789 k kpartx i386 0.4.7-12.el5_1.4 updates + 404 k libvorbis i386 1:1.1.2-3.el5_1.2 updates + 194 k libxslt i386 1.1.17-2.el5_1.1 updates + 485 k mcstrans i386 0.2.7-1.el5 updates + 16 k Transaction Summary ====================================================================== +======= Install 1 Package(s) Update 7 Package(s) Remove 0 Package(s) Total download size: 17 M Is this ok [y/N]: y Downloading Packages: [Fri May 23 10:57:56 2008] (31638) External mirror for /centos/5/updat +es/i386/RPMS/gnutls-1.4.1-3.el5_1.i386.rpm: (1/4): gnutls-1.4.1-3.el5 100% |=========================| 350 kB 0 +0:00 [Fri May 23 10:57:57 2008] (31625) External mirror for /centos/5/updat +es/i386/RPMS/kernel-headers-2.6.18-53.1.21.el5.i386.rpm: (2/4): kernel-headers-2.6 100% |=========================| 789 kB 0 +0:01 [Fri May 23 10:57:59 2008] (31626) External mirror for /centos/5/updat +es/i386/RPMS/libvorbis-1.1.2-3.el5_1.2.i386.rpm: (3/4): libvorbis-1.1.2-3. 100% |=========================| 194 kB 0 +0:00 [Fri May 23 10:57:59 2008] (31627) External mirror for /centos/5/updat +es/i386/RPMS/mcstrans-0.2.7-1.el5.i386.rpm: (4/4): mcstrans-0.2.7-1.e 100% |=========================| 16 kB 0 +0:00 Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : libvorbis ####################### [ 1 +/15] Updating : gnutls ####################### [ 2 +/15] Updating : libxslt ####################### [ 3 +/15] Updating : kpartx ####################### [ 4 +/15] Updating : device-mapper-multipath ####################### [ 5 +/15] Installing: kernel ####################### [ 6 +/15] Updating : kernel-headers ####################### [ 7 +/15] Updating : mcstrans ####################### [ 8 +/15] Cleanup : kpartx ####################### [ 9 +/15] Cleanup : libxslt ####################### [10 +/15] Cleanup : device-mapper-multipath ####################### [11 +/15] Cleanup : gnutls ####################### [12 +/15] Cleanup : kernel-headers ####################### [13 +/15] Cleanup : libvorbis ####################### [14 +/15] Cleanup : mcstrans ####################### [15 +/15] Installed: kernel.i686 0:2.6.18-53.1.21.el5 Updated: device-mapper-multipath.i386 0:0.4.7-12.el5_1.4 gnutls.i386 0 +:1.4.1-3.el5_1 kernel-headers.i386 0:2.6.18-53.1.21.el5 kpartx.i386 0 +:0.4.7-12.el5_1.4 libvorbis.i386 1:1.1.2-3.el5_1.2 libxslt.i386 0:1.1 +.17-2.el5_1.1 mcstrans.i386 0:0.2.7-1.el5 Complete! [Fri May 23 10:58:41 2008] (31638) ERROR: Getting request failed: Clie +nt closed [Fri May 23 10:58:41 2008] (31625) ERROR: Getting request failed: Clie +nt closed [root@vr-mgmt2 centos]# [Fri May 23 10:58:42 2008] (31626) ERROR: Gett +ing request failed: Client closed [Fri May 23 10:58:42 2008] (31627) ERROR: Getting request failed: Clie +nt closed [Fri May 23 10:59:43 2008] (31649) ERROR: Cannot get flock: Interrupte +d system call lf total 20K drwxr-xr-x 4 root root 4.0K Apr 28 10:29 4/ drwxrwxrwx 3 root root 4.0K May 22 01:20 ../ drwxrwxrwx 4 root root 4.0K May 23 10:53 ./ drwxr-xr-x 6 root root 4.0K May 23 10:53 5/ [root@vr-mgmt2 centos]# [Fri May 23 11:00:44 2008] (31631) ERROR: Cann +ot get flock: Interrupted system call lf -R 5 5: total 24K drwxr-xr-x 3 root root 4.0K May 23 10:53 extras/ drwxrwxrwx 4 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 updates/ drwxr-xr-x 3 root root 4.0K May 23 10:53 os/ drwxr-xr-x 3 root root 4.0K May 23 10:53 addons/ drwxr-xr-x 6 root root 4.0K May 23 10:53 ./ 5/extras: total 12K drwxr-xr-x 3 root root 4.0K May 23 10:53 i386/ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ drwxr-xr-x 6 root root 4.0K May 23 10:53 ../ 5/extras/i386: total 12K drwxr-xr-x 2 root root 4.0K May 23 10:53 repodata/ drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/extras/i386/repodata: total 12K -rw-r--r-- 1 root root 1.2K May 23 10:53 repomd.xml drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 2 root root 4.0K May 23 10:53 ./ 5/updates: total 12K drwxr-xr-x 3 root root 4.0K May 23 10:53 i386/ drwxr-xr-x 6 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/updates/i386: total 12K drwxr-xr-x 2 root root 4.0K May 23 10:53 repodata/ drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/updates/i386/repodata: total 12K -rw-r--r-- 1 root root 951 May 23 10:53 repomd.xml drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 2 root root 4.0K May 23 10:53 ./ 5/os: total 12K drwxr-xr-x 3 root root 4.0K May 23 10:53 i386/ drwxr-xr-x 6 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/os/i386: total 12K drwxr-xr-x 2 root root 4.0K May 23 10:53 repodata/ drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/os/i386/repodata: total 12K -rw-r--r-- 1 root root 1.2K May 23 10:53 repomd.xml drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 2 root root 4.0K May 23 10:53 ./ 5/addons: total 12K drwxr-xr-x 3 root root 4.0K May 23 10:53 i386/ drwxr-xr-x 6 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/addons/i386: total 12K drwxr-xr-x 2 root root 4.0K May 23 10:53 repodata/ drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ 5/addons/i386/repodata: total 12K -rw-r--r-- 1 root root 951 May 23 10:53 repomd.xml drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 2 root root 4.0K May 23 10:53 ./ [root@vr-mgmt2 centos]# lf -R 4 4: total 28K drwxr-xr-x 3 root root 4.0K Apr 28 10:29 updates/ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 os/ drwxr-xr-x 4 root root 4.0K Apr 28 10:29 ./ drwxrwxrwx 4 root root 4.0K May 23 10:53 ../ 4/updates: total 24K drwxr-xr-x 2 root root 4.0K Apr 28 10:29 i386/ drwxr-xr-x 4 root root 4.0K Apr 28 10:29 ../ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ./ 4/updates/i386: total 16K drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ../ drwxr-xr-x 2 root root 4.0K Apr 28 10:29 ./ 4/os: total 24K drwxr-xr-x 4 root root 4.0K Apr 28 10:29 ../ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ./ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 i386/ 4/os/i386: total 24K drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ../ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 CentOS/ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ./ 4/os/i386/CentOS: total 24K drwxr-xr-x 2 root root 4.0K Apr 28 10:29 RPMS/ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ../ drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ./ 4/os/i386/CentOS/RPMS: total 16K drwxr-xr-x 3 root root 4.0K Apr 28 10:29 ../ drwxr-xr-x 2 root root 4.0K Apr 28 10:29 ./ [root@vr-mgmt2 centos]# lf total 20K drwxr-xr-x 4 root root 4.0K Apr 28 10:29 4/ drwxrwxrwx 3 root root 4.0K May 22 01:20 ../ drwxrwxrwx 4 root root 4.0K May 23 10:53 ./ drwxr-xr-x 6 root root 4.0K May 23 10:53 5/ [root@vr-mgmt2 centos]# cd 4 [root@vr-mgmt2 4]# cd ../5 [root@vr-mgmt2 5]# lf total 24K drwxr-xr-x 3 root root 4.0K May 23 10:53 extras/ drwxrwxrwx 4 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 updates/ drwxr-xr-x 3 root root 4.0K May 23 10:53 os/ drwxr-xr-x 3 root root 4.0K May 23 10:53 addons/ drwxr-xr-x 6 root root 4.0K May 23 10:53 ./ [root@vr-mgmt2 5]# [Fri May 23 11:01:45 2008] (31625) ERROR: Cannot ge +t flock: Interrupted system call lf updates/ total 12K drwxr-xr-x 3 root root 4.0K May 23 10:53 i386/ drwxr-xr-x 6 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./ [root@vr-mgmt2 5]# lf updates/i386/ total 12K drwxr-xr-x 2 root root 4.0K May 23 10:53 repodata/ drwxr-xr-x 3 root root 4.0K May 23 10:53 ../ drwxr-xr-x 3 root root 4.0K May 23 10:53 ./
So, you can see the yum update succeeded (the second time I ran it). But the only files in the mirror area are .xml files...

-Scott


Comment on Re: Mirror a package repository on the fly.
Download Code
Re^2: Mirror a package repository on the fly.
by motobói (Beadle) on Jun 05, 2008 at 06:45 UTC
    Hum. Interesting.
    Seems like Yum is request headers and package using Partial Content Requests. Let me see a -vvv output.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://688706]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (3)
As of 2014-09-19 21:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (147 votes), past polls