Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

SCP file size greater than 2GB

by shan_emails (Beadle)
on Oct 11, 2013 at 18:01 UTC ( #1057932=perlquestion: print w/ replies, xml ) Need Help??
shan_emails has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I need to SCP the batch of files (*.txt) between two servers which size is greater than 2 GB. How can i handle this scenario. It takes more time to scp the file.

I tried like below.

if size is less than 2GB then go for zip that file (gzip FILENAME) scp FILENAME user@server:/path/. else scp FILENAME user@server:/path/. endif
Is any other optimum solution to handle this scenario.
Any help would be highly appreciated

Thanks,
Shanmugam A.

Comment on SCP file size greater than 2GB
Download Code
Re: SCP file size greater than 2GB
by Happy-the-monk (Monsignor) on Oct 11, 2013 at 18:32 UTC

    have a look at rsync(1) and look for the multitude of compress and zip options there.

    Cheers, Sören

    Créateur des bugs mobiles - let loose once, run everywhere.
    (hooked on the Perl Programming language)

Re: SCP file size greater than 2GB
by hardburn (Abbot) on Oct 11, 2013 at 19:06 UTC

    Does the destination filesystem support files >2GB? Do you need to decompress those files at the destination? If yes to both, then you'll still be stuck. You may be able to transfer them there in compressed form, but you won't be able to decompress them, at least not completely.

    The -C option to scp will compress on the fly.


    "There is no shame in being self-taught, only in not trying to learn in the first place." -- Atrus, Myst: The Book of D'ni.

      Ummmmm? Errrrrrr.... /me scratches head, cogitates, and decides to ask:

      First sentence: did you mean "Doesn't" sted "Does?" Seems to me if remote system ok with files > 2GB, one of the two issues you id'ed is non-existent.... hence, would it be correct to emend s/Does/Doesn't/; and then read the rest of the para as "If the answer to question 1 is 'no' and 'yes' to Q2, you'll still...?"

      Or, :-) , am I having a logical-or mental glitch?

        "Doesn't" sounds wrong to me, but you're right that the first question needs its phrasing fixed. Should be more like:

        Does the destination filesystem not support files >2GB?

        And then fix the end of the paragraph as you described.

        I'm not happy with the negation there, but I don't feel like taking the time for a more careful fix at the moment :)


        "There is no shame in being self-taught, only in not trying to learn in the first place." -- Atrus, Myst: The Book of D'ni.

Re: SCP file size greater than 2GB
by rminner (Hermit) on Oct 11, 2013 at 23:11 UTC
    happy-the-monk already named a good option with rsync. I am just going to throw another one into the room. Perhaps it will be of some help to you. My proposal will however compress all files, no matter the size.
    I like to copy files using the combination of tar and ssh. Like that you just pipe one big bytestream through your connection (and you can transparently add compression). If all the files you want to copy are in one directory (and no others), you could try the following:
    tar czf - SOURCE_DIR | ssh USER@REMOTE_HOST "cd REMOTE_DIR; tar xzf -" # or if the tar is sufficient on the other side: tar czf - SOURCE_DIR | ssh USER@REMOTE_HOST "cat >REMOTE_DIR/backup.tg +z"
    In my experience the combination of tar and ssh is much faster than scp if you are copying many small files. If your system is more current, your tar might also provide different compression methods like -j or -J:
    -j, --bzip2 filter the archive through bzip2 -J, --xz filter the archive through xz -z, --gzip, --gunzip --ungzip filter the archive through gzip
    bzip2 and xz will usually reach higher compression levels than gzip. They require however more cpu and ram, and the compression will be slower. In my experience one can achieve the highest compression with xz (-J). If space/bandwidth matters more than (cpu-/real-)time and RAM, you might try stronger compression methods. For most cases gzip will however also be good enough and quite fast.
    If you just want to copy a few selected files from random locations you could for example create a file with the filenames and then pass it to tar using -T:
    -T, --files-from FILE get names to extract or create from FILE
    To skip files, use -X:
    -X, --exclude-from FILE exclude patterns listed in FILE
    If you want a graphical progress you can combine it with pv (you might have to install it first). E.g.:
    tar czf - SOURCE_DIR | pv | ssh USER@REMOTE_HOST "cd REMOTE_DIR; tar x +zf -" # or if the tar is sufficient on the other side: tar czf - SOURCE_DIR | pv | ssh USER@REMOTE_HOST "cat >REMOTE_DIR/back +up.tgz"

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1057932]
Approved by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (10)
As of 2014-09-19 10:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (135 votes), past polls