perlquestion
longjohnsilver
<p> Good Day to all of you Enlightened Monks,<br><br>
I've been trying to find the best solution to perform some simple yet complicate (performance-wise) tasks on a "busy" NFS file system.<br><br>
Let me be clearer: I'm on a Linux Debian Box and the filesystem is mounted as follows:<br><br>
/mnt/mounted_dir/ type nfs (rw,noatime,rsize=32768,wsize=32768,hard,intr,tcp,nfsvers=3,addr=x.x.x.x)
<br><br>
By "busy" i mean that at "random" times each day a lot of (maybe 15 x second) small .xml files (~2k each) get written on this filesystem.
<br><br>
The things i need to perform on these files are the following:<br><br>
1. Read/Fetch each file as soon as it has been written completely.<br>
2. SFTP-Put the file on a remote FTP server.<br>
3. Make sure the transfer has been completed successfully<br>
4. Move the local copy of the "transferred" file to another local directory.<br><br>
These tasks don't look really complicated at first sight but, i have to say, that when the filesystem finds itself in this "busy" state
even executing a simple "ls" takes ages, really ages. The script only works smoothly when the filesystem is not in a "busy" state.
<br><br>
Here's the core snippet of my code for anyone interested to helping me out in making things perform better. Thanks.
<br><br>
<code>
#!/usr/bin/perl -w
use strict;
use diagnostics;
use autodie;
use Net::SFTP::Foreign;
use File::Copy;
use File::stat;
-->> omitted code to make things more readable
$sftp = Net::SFTP::Foreign->new($host, %args);
$sftp->setcwd($remote_dir) || die log_msg($sftp->error."Exiting...\n");
opendir($DH, $local_dir) or die $!;
while (defined(my $file = readdir($DH)) {
my $mtime = stat("$local_dir/$file")->mtime;
my $age =(time - $mtime);
chomp($age);
next unless ($age > 2);
next unless (-f "$local_dir/$file");
next unless ($file =~ m/\.xml$/);
# sftp put section
if ($sftp->put("$local_dir/$file")) {
move("$local_dir/$file", "$local_dir_mv") or die log_msg("The move operation failed: $!");
}
else {
die log_msg($sftp->error);
}
}
closedir($DH);
</code>