Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

SCP file size greater than 2GB

by shan_emails (Beadle)
on Oct 11, 2013 at 18:01 UTC ( #1057932=perlquestion: print w/replies, xml ) Need Help??
shan_emails has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I need to SCP the batch of files (*.txt) between two servers which size is greater than 2 GB. How can i handle this scenario. It takes more time to scp the file.

I tried like below.

if size is less than 2GB then go for zip that file (gzip FILENAME) scp FILENAME user@server:/path/. else scp FILENAME user@server:/path/. endif
Is any other optimum solution to handle this scenario.
Any help would be highly appreciated

Shanmugam A.

Replies are listed 'Best First'.
Re: SCP file size greater than 2GB
by Happy-the-monk (Abbot) on Oct 11, 2013 at 18:32 UTC

    have a look at rsync(1) and look for the multitude of compress and zip options there.

    Cheers, Sören

    Créateur des bugs mobiles - let loose once, run everywhere.
    (hooked on the Perl Programming language)

Re: SCP file size greater than 2GB
by rminner (Hermit) on Oct 11, 2013 at 23:11 UTC
    happy-the-monk already named a good option with rsync. I am just going to throw another one into the room. Perhaps it will be of some help to you. My proposal will however compress all files, no matter the size.
    I like to copy files using the combination of tar and ssh. Like that you just pipe one big bytestream through your connection (and you can transparently add compression). If all the files you want to copy are in one directory (and no others), you could try the following:
    tar czf - SOURCE_DIR | ssh USER@REMOTE_HOST "cd REMOTE_DIR; tar xzf -" # or if the tar is sufficient on the other side: tar czf - SOURCE_DIR | ssh USER@REMOTE_HOST "cat >REMOTE_DIR/ +z"
    In my experience the combination of tar and ssh is much faster than scp if you are copying many small files. If your system is more current, your tar might also provide different compression methods like -j or -J:
    -j, --bzip2 filter the archive through bzip2 -J, --xz filter the archive through xz -z, --gzip, --gunzip --ungzip filter the archive through gzip
    bzip2 and xz will usually reach higher compression levels than gzip. They require however more cpu and ram, and the compression will be slower. In my experience one can achieve the highest compression with xz (-J). If space/bandwidth matters more than (cpu-/real-)time and RAM, you might try stronger compression methods. For most cases gzip will however also be good enough and quite fast.
    If you just want to copy a few selected files from random locations you could for example create a file with the filenames and then pass it to tar using -T:
    -T, --files-from FILE get names to extract or create from FILE
    To skip files, use -X:
    -X, --exclude-from FILE exclude patterns listed in FILE
    If you want a graphical progress you can combine it with pv (you might have to install it first). E.g.:
    tar czf - SOURCE_DIR | pv | ssh USER@REMOTE_HOST "cd REMOTE_DIR; tar x +zf -" # or if the tar is sufficient on the other side: tar czf - SOURCE_DIR | pv | ssh USER@REMOTE_HOST "cat >REMOTE_DIR/back +up.tgz"
Re: SCP file size greater than 2GB
by hardburn (Abbot) on Oct 11, 2013 at 19:06 UTC

    Does the destination filesystem support files >2GB? Do you need to decompress those files at the destination? If yes to both, then you'll still be stuck. You may be able to transfer them there in compressed form, but you won't be able to decompress them, at least not completely.

    The -C option to scp will compress on the fly.

    "There is no shame in being self-taught, only in not trying to learn in the first place." -- Atrus, Myst: The Book of D'ni.

      Ummmmm? Errrrrrr.... /me scratches head, cogitates, and decides to ask:

      First sentence: did you mean "Doesn't" sted "Does?" Seems to me if remote system ok with files > 2GB, one of the two issues you id'ed is non-existent.... hence, would it be correct to emend s/Does/Doesn't/; and then read the rest of the para as "If the answer to question 1 is 'no' and 'yes' to Q2, you'll still...?"

      Or, :-) , am I having a logical-or mental glitch?

        "Doesn't" sounds wrong to me, but you're right that the first question needs its phrasing fixed. Should be more like:

        Does the destination filesystem not support files >2GB?

        And then fix the end of the paragraph as you described.

        I'm not happy with the negation there, but I don't feel like taking the time for a more careful fix at the moment :)

        "There is no shame in being self-taught, only in not trying to learn in the first place." -- Atrus, Myst: The Book of D'ni.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1057932]
Approved by Old_Gray_Bear
[choroba]: also, I discovered several vegan/vegetarian restaurants, so I can avoid meat
[LanX]: ha ha ... happy Czech :)
[choroba]: been to the War Remnants Museum, too. Quite a brutal experience
[choroba]: but still, small-talk about smuggling guns and how Trump is great with US expats isn't my cup of tea
[LanX]: brutal beer?
[LanX]: met a Canadian yesterday, he suggested building a wall and let the US pay for it
[choroba]: No, but lots of pictures of the Agent Orange victims, plus some deformed foeti
[LanX]: ugh

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (6)
As of 2017-02-26 04:39 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (371 votes). Check out past polls.