Copying a list of files from a directory to a new directory

[ramjamman]

Hi Monks I would welcome any suggestions as to the best way to resolve this problem

I have on a centos server about 7000 files mostly pdf in one directory. I need to download about 4000 of them for which I have a list in a csv or txt file of their names.

I am thinking the best way to approach this is to run a script from the command line to copy the files required into a new directory using a script something like that below and then I can download them .

However I am always open to any suggestions of any other methods that would achieve a solution to this problem Thank You

#!/usr/bin/perl -w
open FILE , '<' , "mylist.txt" || die "$!";
while (<FILE>){
        chomp;
        system("cp $_ newdirectory/");
}
close FILE;

[roboticus]

ramjamman:

If you're on a *NIX box and have a text file containing a list of files to copy to a new location, you don't even need to use perl to do the copy. Instead, you can use xargs and tar to do the job. As an example, here's a text file with a list of files to copy:

$ cat mylist.txt
foo
bar
bim/bam

And in our source directory, we have the specified files, along with a file we want to ignore:

$ ls -alR source
source:
total 3.0K
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:51 ./
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:49 ../
-rw-r--r--  1 Roboticus None 4 Jul 21 00:47 bar
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:48 bim/
-rw-r--r--  1 Roboticus None 8 Jul 21 00:47 foo
-rw-r--r--  1 Roboticus None 7 Jul 21 00:51 ignore

source/bim:
total 1.0K
drwxr-xr-x+ 1 Roboticus None  0 Jul 21 00:48 ./
drwxr-xr-x+ 1 Roboticus None  0 Jul 21 00:51 ../
-rw-r--r--  1 Roboticus None 13 Jul 21 00:48 bam

At this time, our dest directory doesn't have anything interesting in it:

$ ls -alR dest
dest:
total 0
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:51 ./
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:49 ../

So all we need to do is to start a process, change to the source directory and then use xargs to tell tar which files to copy into an archive. Then we'll pipe that archive to another process wherein we change to the destination directory and extract all the files, like this:

$ (cd source; xargs <../mylist.txt tar cf -) | (cd dest; tar xf -)

After running that, the files we wanted to copy to dest are there:

$ ls -alR dest
dest:
total 2.0K
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:52 ./
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:49 ../
-rw-r--r--  1 Roboticus None 4 Jul 21 00:47 bar
drwxr-xr-x+ 1 Roboticus None 0 Jul 21 00:52 bim/
-rw-r--r--  1 Roboticus None 8 Jul 21 00:47 foo

dest/bim:
total 1.0K
drwxr-xr-x+ 1 Roboticus None  0 Jul 21 00:52 ./
drwxr-xr-x+ 1 Roboticus None  0 Jul 21 00:52 ../
-rw-r--r--  1 Roboticus None 13 Jul 21 00:48 bam

...roboticus

When your only tool is a hammer, all problems look like your thumb.

[Laurent_R]

Yes, roboticus++, you're right, a one-line shell command at the prompt is sufficient for the task at hand.

But not being a great fan of the shell syntax, I would personally tend to do it with a Perl one-liner, something like this (reusing here the OP's syntax):

perl -ne 'chomp; system("cp $_ newdirectory/")' mylist.txt

even though it might be slightly less efficient, or possibly something similar using the File::Copy module.

[soonix]

The -f - is redundant, AFAIK. Then, newer (at least GNU) tar has a -T option (or --files-from) which eliminates the need for xargs, and a -C option (or --directory) which eliminates the need for cd and the parentheses

This simplifies the script to

tar c -C source -T mylist.txt | tar xC dest

(Tested with MinGW on Win7)

Of course, your script could be written right out of your head, while I had to open the man page for looking up the letter of that filelist option...

[roboticus]

soonix:

Heh, heh! In fact, when I was writing that response, I thought that tar would have such an option. But after listing through the first three or so pages of the dox, I got tired of looking. Then I thought I'd just xargs to convert the file into a list of command-line arguments. I nearly wrote it as cat mylist.txt | xargs | ... but didn't want to earn another "useless use of cat" award, so I just used xargs <mylist.txt to do the task.

After your response, I again looked at man tar and found the option ... on the thirteenth page! Since my terminal is 57 lines long, that seems a bit excessive. I'm thinking that the man page for tar ought to be rearranged a bit, as that sounds like an option that's generally useful, and there are quite a few options that seem to be much more specialized, and hence useful in fewer situations. (I've never needed --hole-detection or --check-device both of which are on page 4.) Now that I'm aware of -T, though, I'll hopefully remember it for the next time. ;^)

...roboticus

When your only tool is tar, all problems look like you're gonna have to read "War and Peace", possibly several times.

[ramjamman]

The wisdom available here never ceases to impress me !!

Thank you so much for your replies

[AnomalousMonk]

open FILE , '<' , "mylist.txt" || die "$!";

Not a direct answer to your question, but an incidental thought. The above quoted statement should be either
    open FILE , '<' , "mylist.txt" or die "$!";
or else
    open(FILE , '<' , "mylist.txt") || die "$!";
because the expression  "mylist.txt" || die "$!" as quoted will be evaluated before the open happens, and the particular file name in the example (and almost any other file name) evaluates as true (i.e., it's not 0, '0', '' or undef; see What is true and false in Perl?), so the call to die will never be made no matter what happens with the open call. (Update: And see perlop for info on Operator Precedence and Associativity; the above is a precedence problem.)

P.S.: I tend to prefer the lexical filehandle syntax
    my $filename = 'mylist.txt';
    open my $filehandle, '<', $filename or die "opening '$filename': $!";
(update: or else maybe useautodie;).

Update: I've written above that almost any file name you can think of will evaluate as true in Perl. But let's say you actually had a file named '0'. In this case, what would happen given the quoted statement?

[Laurent_R]

You might want to take a look at the File::Copy module.