SFTP->RPUT Crashes

[joshpc99]

I'm semi-new to Perl, I haven't used it in years. I'm trying to write a basic script to transfer large amounts of data from a windows 7 machine to a linux server. Below is a brief portion of my code (some bits removed but I don't believe they are related to my issues). After much searching, I decided to go with SFTP->FOREIGN with the BACKEND into SSH2. Large in part, everything works great. All of my testing worked liked a charm. However, these tests used small amounts of data (~50MB). My script is set to run nightly at midnight and transfer any new folders on windows to linux. Each folder could be up to 20GB or more. Right now, there are 6 or 7 folders waiting to transfer. I've run it several times and most of the files are transferred, but never all. It'll go through the first folder and copy ~70% - different each time and crash before it has a chance to get the next folder. I can't get any error codes or debug information. No matter what I try, a new cmd windows pops up with an error and it crashes and closes automatically before I can even see it. Getting very frustrating. It looks like I may have to write my own recursion and use the regular 'put' instead of 'rput'. Any ideas?

#!\c:\perl

    use Net::SSH2;
    use Net::SFTP::Foreign;
    use Net::SFTP::Recursive;
    use File::Copy::Recursive qw (dircopy);
    use File::Find;
    use File::Path;
    use Cwd;

    use strict;
    use warnings;



        if (opendir(DIR, $SOURCE2)){
            while (defined ($file = readdir(DIR))) {
                if (($file eq ".") || ($file eq "..")){
                }

                elsif (-d "$SOURCE2/$file"){
                    if (-e "$SOURCE2/$file/Transferred to $HOST.txt"){
                        if (int(-M "$SOURCE2/$file/Transferred to $HOS
+T.txt") > $life) {
                            rmtree ("$SOURCE2/$file");
                        }
                    }
                    elsif (-e "$SOURCE2/$file/RTAComplete.txt") {
                        if (($sftp->rput("$SOURCE2/$file", "$PATH/$fil
+e", late_set_perm => 1, overwrite => 0) or die "Rput died: ".$sftp->e
+rror) > 0){
                            open FH, ">$SOURCE2/$file/Transferred to $
+HOST.txt";
                            print FH "";
                            close FH;
                        }
                    }
                }
            }
            closedir(DIR);
        }

[Eliya]

In case the "crash" is related to a die from somewhere within the code, you could use eval {} to catch the error.  If, OTOH, it terminates with a segfault, or bus error, or something, it would be more difficult to debug.  (If it was on Unix, I'd say set things up to get a core file to load into a debugger — but on Windows, I don't know how to get a core file...  maybe someone else does(?)).

Update: as an alternative, you might want to look into Unison instead — it syncs folders cross-platform in a convenient way.

[salva]

rput does not fails on errors, it just ignores then, but you can handle then using the on_error callback.

To get debugging information from Net::SFTP::Foreign, at the beginning of your script, redirect STDERR to some file and activate the following debugging flags:

open STDERR, '>', 'c:\\sftp.debug';
$Net::SFTP::Foreign::debug = (2|4|64|256|4096|8192|32768);

You can send me the output to the email that appears on Net::SFTP::Foreign documentation (I am its author) for inspection.

IMO, the most likely cause of the problem may be some bug in Net::SSH2, ensure you are using the latest version compiled also against the latest version of libssh2 as both are continuosly being improved.

[JavaFan]

I cannot answer the question why your program is crashing, but it looks to me you're trying to re-implement rsync. Why not use that?

[joshpc99]

Thanks for all of the ideas. Comments on each:

@Eliya - Unrelated to the 'or die' portion. It crashed before this, I only added it to try to debug - nothing has ever printed with it. Never heard of Unison but if all of this fails, I'll give that a shot.

@JavaFan - Never heard of rsync either but it also looks promising.

@salva - You're right about rput not failing. I just updated Perl, SSH2, Foreign, and the Backend and no change. I have some output from the code you suggested, I'll send it your way shortly. Some interesting info: I ran it twice after trying to make small changes. While the file size is the exact same and the same number of lines output, the files described are not the same - that I can see of. I hope you can make some sense of it.

[oko1]

I'm with JavaFan on this one; stick with 'rsync'. Unison is more of a distributed filesystem (although they claim otherwise) - and pretty much runs on 'rsync' anyway. It also requires a good bit of configuration. With 'rsync', simply launching it from the command line with reasonable options will do everything you need. E.g., something like this:

rsync -av -e "ssh -l ssh-user" /source host:/dest

It can do updates (I love that capability...) as well as full backups, progress bars, comparisons between file sets, etc. One of the most useful programs ever, IMO.

[dwm042]

One of the practical issues with large volume SFTP is that if your network isn't clean, it is pretty much guaranteed to fail somewhere during the transmission.

Any transmission protocol that relies on a bunch of really large files to successfully complete in sequence is a failure in design.

Better would be a design that 'put' files individually, and then would retransmit if individual files fail. The protocol then works through networking issues via sheer dogged persistence.

My 0.02 anyway. This isn't a knock on 'rput', which would work if your files were small, but more a reality check on the issues with transmitting many gigabytes sans error.

David.

[salva]

One of the practical issues with large volume SFTP is that if your network isn't clean, it is pretty much guaranteed to fail somewhere during the transmission

That's plain wrong.

Just now, for instance, on my desktop computer, physically placed in Spain, I have the file system of a remote machine in Mexico (9000 km from here) mounted via sshfs.

It has been running non-stop for more than two weeks without a single issue, and I had used it to transfer several GB of files in both directions. The connection goes through two gateways and over an VPN that has been restarted a couple of times, but the TCP connection has survived all of this.

TCP was designed to work on the unreliable networks of 30 years ago, and today, the conditions even on the worst cases are far better than what they used to be.

If you have problems transmitting some GB of data reliably, then, either you are using defective network equipment or your network technicians are not very capable.