Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

FTP: Unable to close datastream

by m0uthless (Initiate)
on Feb 19, 2013 at 18:28 UTC ( #1019639=perlquestion: print w/replies, xml ) Need Help??
m0uthless has asked for the wisdom of the Perl Monks concerning the following question:

I'm approaching the world of perl these days. I created a script that would automatically download from an FTP new files modified in the last 24 hours and with .tar extension, but unfortunately the script fails. This is the script
use strict; use warnings; use Net::FTP; use DateTime (); use DateTime::Duration (); use File::Listing qw(parse_dir); use POSIX qw(strftime); use File::stat; my $pathto = "/mnt/tmp"; my $logname = "_log"; my ($host, $user, $passwd) = ('IP.ADD.RE.SS', 'user', 'password'); my $dir = '/common'; my $timesplit = 24; my $extension = '.tar'; my @noseries = ('monkey','dog','rubbit'); sub findstring { foreach my $series(@noseries) { if (index((lc $_[0]),(lc $series)) != -1) { return 1; } } return 0; } sub printclose { print MYOUTFILE localtime()."\: $_[0]\n"; close(MYOUTFILE); return "$_[0]\n"; } sub printfile { print MYOUTFILE localtime()."\: $_[0]\n"; return "$_[0]\n"; } my $now = DateTime->now; my $delta = DateTime::Duration->new( hours => $timesplit ); my $cutoff = $now->subtract_duration( $delta ); open(MYOUTFILE, '>>'.$pathto.$logname) or die "Impossibile creare o aprire il file di log"; #open for wri +te, append print printfile('Avvio operazione'); print printfile('Mi connetto al server'); my $ftp = Net::FTP->new($host, Debug=> 1, Passive => 1 ) or die printclose("Non riesco a connettermi a $host"); print printfile('Effettuo il login'); $ftp->login($user, $passwd) or die printclose('Cannot login'); $ftp->binary(); print printfile('Seleziono la directory'); $ftp->cwd($dir) or die printclose('Cannot cwd to $dir'); print printfile('Cerco file da scaricare'); my $ls = $ftp->dir() or die printclose('Cannot list $dir'); foreach my $entry (parse_dir($ls)) { my ($name, $type, $size, $mtime, $mode) = @$entry; next unless ($type eq 'f' && ((lc (substr $name, -4)) cmp $extensi +on) == 0 ); my $tz = DateTime::TimeZone->new( name => 'local' ); ( my $sec,my $min,my $hour,my $mday,my $months,my $year,my $wday,m +y $yday,my $isdst) = localtime($mtime); $year= $year+1900; $months=$months+1; my $dt = DateTime->new(year => $year, month => $months, day => $m +day, hour => $hour, minute => $min, second => $min, time_zone => Date +Time::TimeZone::Local->TimeZone()); my $cmp = DateTime->compare( $dt, $cutoff ); if ($cmp >=0) { if (findstring($name)==0) { print (printfile("Downloading $name")); $ftp->get($name,"$pathto$name")|| die printclose("get($name) failed: ". $ftp->code() . ": " +. $ftp->message()); my $filesize = stat("$pathto$name")->size; if ($filesize!=$size) { print (printfile("Attenzione: File scaricato ma il siz +e non corrisponde, possibile errore nel download")); } else { print (printfile("Scaricato con successo")); } } } } $ftp->quit; print printclose('Operazione completata con successo');
While this is the debug
root@pc2:~# perl Avvio operazione Mi connetto al server Net::FTP>>> Net::FTP(2.77) Net::FTP>>> Exporter(5.67) Net::FTP>>> Net::Cmd(2.29) Net::FTP>>> IO::Socket::INET(1.33) Net::FTP>>> IO::Socket(1.34) Net::FTP>>> IO::Handle(1.33) Net::FTP=GLOB(0x30ef478)<<< 220-FileZilla Server version 0.9.41 beta Net::FTP=GLOB(0x30ef478)<<< 220-written by Tim Kosse ( +) Net::FTP=GLOB(0x30ef478)<<< 220 Please visit +ojects/filezilla/ Effettuo il login Net::FTP=GLOB(0x30ef478)>>> USER user Net::FTP=GLOB(0x30ef478)<<< 331 Password required for user Net::FTP=GLOB(0x30ef478)>>> PASS .... Net::FTP=GLOB(0x30ef478)<<< 230 Logged on Net::FTP=GLOB(0x30ef478)>>> TYPE I Net::FTP=GLOB(0x30ef478)<<< 200 Type set to I Seleziono la directory Net::FTP=GLOB(0x30ef478)>>> CWD /common Net::FTP=GLOB(0x30ef478)<<< 250 CWD successful. "/common" is currentdi +rectory. Cerco file da scaricare Net::FTP=GLOB(0x30ef478)>>> PASV Net::FTP=GLOB(0x30ef478)<<< 227 Entering Passive Mode (IP.ADD.RE.SS,82 +,8) Net::FTP=GLOB(0x30ef478)>>> LIST Net::FTP=GLOB(0x30ef478)<<< 150 Connection accepted Net::FTP=GLOB(0x30ef478)<<< 226 Transfer OK Downloading mainupdate_20130218.tar Net::FTP=GLOB(0x30ef478)>>> PASV Net::FTP=GLOB(0x30ef478)<<< 227 Entering Passive Mode (IP.ADD.RE.SS,82 +,9) Net::FTP=GLOB(0x30ef478)>>> RETR mainupdate_20130218.tar Net::FTP=GLOB(0x30ef478)<<< 150 Connection accepted Net::FTP=GLOB(0x30ef478): Timeout at /usr/share/perl/5.10/Net/FTP/data line 74. Unable to close datastream at /root/ line 73. get(mainupdate_20130218.tar) failed: 150: Connection accepted
Thanks, Logan

Replies are listed 'Best First'.
Re: FTP: Unable to close datastream
by Illuminatus (Curate) on Feb 19, 2013 at 19:51 UTC
    First, thank you for taking the time to read the posting FAQs so that your question is clear. OK, now to your problem. I think the 'unable to close datastream' is a red-herring. The message before that indicates that the transfer timed out. It's not clear from the debug output how much time passed between when the transfer started and when it reported a timeout. Here is what I would try:
    1. Make sure you can execute these commands using a manual ftp client.
    2. If the file transfers manually, but the connection is slow, you can bump the timeout up. It defaults to 120 seconds
    3. Do you have to use passive mode? Will it work in active?
    4. If you have permission, turn up debugging on the ftp server. It might give you more info on what is failing


      Thx for the reply.

      1) with ftp client works, and this script also works with smaller files but not with big ones (> 1 gb)

      2) First of all the downloaded file even with the error is ok. I tried to increase the timeout, but for example if i set 1000 seconds even if the download it's already finished the procedure wait 1000 seconds and then throws the error.

      3) If I disable passive mode when I run the script it stops at $ftp->dir() (if i run it on windows 7 it works well even in active mode);

      4) It doesn't seems there is an advanced debug/log in filezilla server, but to the server the transfer results finished and successfully (226 Transfer OK)

      Thanks, Logan

Re: FTP: Unable to close datastream
by runrig (Abbot) on Feb 05, 2016 at 00:43 UTC
    FWIW, I was running into this very same issue. On large files from one ftp server, I'd get a "Unable to close datastream" warning (which, since 'get' returned undef, was interpreted as an error), yet, the entire file was downloaded. I just ended up working around the problem, making sure local and remote file sizes matched.

    Also, command line ftp seemed to download fine without error or warning.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1019639]
Approved by Corion
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (3)
As of 2017-06-28 02:03 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (619 votes). Check out past polls.