Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Good Day to all of you Enlightened Monks,

I've been trying to find the best solution to perform some simple yet complicate (performance-wise) tasks on a "busy" NFS file system.

Let me be clearer: I'm on a Linux Debian Box and the filesystem is mounted as follows:

/mnt/mounted_dir/ type nfs (rw,noatime,rsize=32768,wsize=32768,hard,intr,tcp,nfsvers=3,addr=x.x.x.x)

By "busy" i mean that at "random" times each day a lot of (maybe 15 x second) small .xml files (~2k each) get written on this filesystem.

The things i need to perform on these files are the following:

1. Read/Fetch each file as soon as it has been written completely.
2. SFTP-Put the file on a remote FTP server.
3. Make sure the transfer has been completed successfully
4. Move the local copy of the "transferred" file to another local directory.

These tasks don't look really complicated at first sight but, i have to say, that when the filesystem finds itself in this "busy" state even executing a simple "ls" takes ages, really ages. The script only works smoothly when the filesystem is not in a "busy" state.

Here's the core snippet of my code for anyone interested to helping me out in making things perform better. Thanks.

#!/usr/bin/perl -w use strict; use diagnostics; use autodie; use Net::SFTP::Foreign; use File::Copy; use File::stat; -->> omitted code to make things more readable $sftp = Net::SFTP::Foreign->new($host, %args); $sftp->setcwd($remote_dir) || die log_msg($sftp->error."Exiting...\n") +; opendir($DH, $local_dir) or die $!; while (defined(my $file = readdir($DH)) { my $mtime = stat("$local_dir/$file")->mtime; my $age =(time - $mtime); chomp($age); next unless ($age > 2); next unless (-f "$local_dir/$file"); next unless ($file =~ m/\.xml$/); # sftp put section if ($sftp->put("$local_dir/$file")) { move("$local_dir/$file", "$local_dir_mv") or die log_msg("Th +e move operation failed: $!"); } else { die log_msg($sftp->error); } } closedir($DH);

In reply to Read, SFTP Put and Move Files from a "Busy" NFS FileSystem by longjohnsilver

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (6)
As of 2023-01-30 12:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?