Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

SFTP->RPUT Crashes

by joshpc99 (Initiate)
on Feb 18, 2012 at 07:01 UTC ( #954712=perlquestion: print w/ replies, xml ) Need Help??
joshpc99 has asked for the wisdom of the Perl Monks concerning the following question:

I'm semi-new to Perl, I haven't used it in years. I'm trying to write a basic script to transfer large amounts of data from a windows 7 machine to a linux server. Below is a brief portion of my code (some bits removed but I don't believe they are related to my issues). After much searching, I decided to go with SFTP->FOREIGN with the BACKEND into SSH2. Large in part, everything works great. All of my testing worked liked a charm. However, these tests used small amounts of data (~50MB). My script is set to run nightly at midnight and transfer any new folders on windows to linux. Each folder could be up to 20GB or more. Right now, there are 6 or 7 folders waiting to transfer. I've run it several times and most of the files are transferred, but never all. It'll go through the first folder and copy ~70% - different each time and crash before it has a chance to get the next folder. I can't get any error codes or debug information. No matter what I try, a new cmd windows pops up with an error and it crashes and closes automatically before I can even see it. Getting very frustrating. It looks like I may have to write my own recursion and use the regular 'put' instead of 'rput'. Any ideas?

#!\c:\perl use Net::SSH2; use Net::SFTP::Foreign; use Net::SFTP::Recursive; use File::Copy::Recursive qw (dircopy); use File::Find; use File::Path; use Cwd; use strict; use warnings; if (opendir(DIR, $SOURCE2)){ while (defined ($file = readdir(DIR))) { if (($file eq ".") || ($file eq "..")){ } elsif (-d "$SOURCE2/$file"){ if (-e "$SOURCE2/$file/Transferred to $HOST.txt"){ if (int(-M "$SOURCE2/$file/Transferred to $HOS +T.txt") > $life) { rmtree ("$SOURCE2/$file"); } } elsif (-e "$SOURCE2/$file/RTAComplete.txt") { if (($sftp->rput("$SOURCE2/$file", "$PATH/$fil +e", late_set_perm => 1, overwrite => 0) or die "Rput died: ".$sftp->e +rror) > 0){ open FH, ">$SOURCE2/$file/Transferred to $ +HOST.txt"; print FH ""; close FH; } } } } closedir(DIR); }

Comment on SFTP->RPUT Crashes
Download Code
Re: SFTP->RPUT Crashes
by Eliya (Vicar) on Feb 18, 2012 at 10:16 UTC

    In case the "crash" is related to a die from somewhere within the code, you could use eval {} to catch the error.  If, OTOH, it terminates with a segfault, or bus error, or something, it would be more difficult to debug.  (If it was on Unix, I'd say set things up to get a core file to load into a debugger — but on Windows, I don't know how to get a core file...  maybe someone else does(?)).

    Update: as an alternative, you might want to look into Unison instead — it syncs folders cross-platform in a convenient way.

Re: SFTP->RPUT Crashes
by JavaFan (Canon) on Feb 18, 2012 at 11:03 UTC
    I cannot answer the question why your program is crashing, but it looks to me you're trying to re-implement rsync. Why not use that?
Re: SFTP->RPUT Crashes
by salva (Monsignor) on Feb 18, 2012 at 11:35 UTC
    rput does not fails on errors, it just ignores then, but you can handle then using the on_error callback.

    To get debugging information from Net::SFTP::Foreign, at the beginning of your script, redirect STDERR to some file and activate the following debugging flags:

    open STDERR, '>', 'c:\\sftp.debug'; $Net::SFTP::Foreign::debug = (2|4|64|256|4096|8192|32768);

    You can send me the output to the email that appears on Net::SFTP::Foreign documentation (I am its author) for inspection.

    IMO, the most likely cause of the problem may be some bug in Net::SSH2, ensure you are using the latest version compiled also against the latest version of libssh2 as both are continuosly being improved.

Re: SFTP->RPUT Crashes
by joshpc99 (Initiate) on Feb 18, 2012 at 21:02 UTC

    Thanks for all of the ideas. Comments on each:

    @Eliya - Unrelated to the 'or die' portion. It crashed before this, I only added it to try to debug - nothing has ever printed with it. Never heard of Unison but if all of this fails, I'll give that a shot.

    @JavaFan - Never heard of rsync either but it also looks promising.

    @salva - You're right about rput not failing. I just updated Perl, SSH2, Foreign, and the Backend and no change. I have some output from the code you suggested, I'll send it your way shortly. Some interesting info: I ran it twice after trying to make small changes. While the file size is the exact same and the same number of lines output, the files described are not the same - that I can see of. I hope you can make some sense of it.

Re: SFTP->RPUT Crashes
by oko1 (Deacon) on Feb 18, 2012 at 21:47 UTC

    I'm with JavaFan on this one; stick with 'rsync'. Unison is more of a distributed filesystem (although they claim otherwise) - and pretty much runs on 'rsync' anyway. It also requires a good bit of configuration. With 'rsync', simply launching it from the command line with reasonable options will do everything you need. E.g., something like this:

    rsync -av -e "ssh -l ssh-user" /source host:/dest

    It can do updates (I love that capability...) as well as full backups, progress bars, comparisons between file sets, etc. One of the most useful programs ever, IMO.

    I hate storms, but calms undermine my spirits.
     -- Bernard Moitessier, "The Long Way"
Re: SFTP->RPUT Crashes
by dwm042 (Priest) on Feb 20, 2012 at 20:49 UTC
    One of the practical issues with large volume SFTP is that if your network isn't clean, it is pretty much guaranteed to fail somewhere during the transmission.

    Any transmission protocol that relies on a bunch of really large files to successfully complete in sequence is a failure in design.

    Better would be a design that 'put' files individually, and then would retransmit if individual files fail. The protocol then works through networking issues via sheer dogged persistence.

    My 0.02 anyway. This isn't a knock on 'rput', which would work if your files were small, but more a reality check on the issues with transmitting many gigabytes sans error.


      One of the practical issues with large volume SFTP is that if your network isn't clean, it is pretty much guaranteed to fail somewhere during the transmission

      That's plain wrong.

      Just now, for instance, on my desktop computer, physically placed in Spain, I have the file system of a remote machine in Mexico (9000 km from here) mounted via sshfs.

      It has been running non-stop for more than two weeks without a single issue, and I had used it to transfer several GB of files in both directions. The connection goes through two gateways and over an VPN that has been restarted a couple of times, but the TCP connection has survived all of this.

      TCP was designed to work on the unreliable networks of 30 years ago, and today, the conditions even on the worst cases are far better than what they used to be.

      If you have problems transmitting some GB of data reliably, then, either you are using defective network equipment or your network technicians are not very capable.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://954712]
Approved by Corion
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2014-08-02 05:01 GMT
Find Nodes?
    Voting Booth?

    Who would be the most fun to work for?

    Results (54 votes), past polls