Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

File copier

by TechFly (Scribe)
on Jul 20, 2010 at 19:13 UTC ( #850494=CUFP: print w/replies, xml ) Need Help??

I have been learning Perl, so if this is too simple I am sorry. I needed a way to copy the file outputs of the nmon script I run on AIX machines. I wanted to move them to a single machine where there is plenty of disk for it. This is what I came up with.

#!/usr/bin/perl -w =head1 Author: TechFly Name: Description: Copy all nmon files from the /tmp/nmonout folders on the +machiens listed in the machinelistfilename. This will create a direc +tory on the destination if needed, and will use SCP to copy the files +. Then it will use SSH and delete the files (less two days from runt +ime) from the source. Start date: 7-20-2010 Last updated Date: 7-20-2010 =cut use strict; use warnings; use Net::SCP qw(scp); use Net::SSH qw(ssh); my $scp; my $machinelist; my $machinelistfilename = "./machinelist.txt"; my $server; open($machinelist, "<", $machinelistfilename)||die $!; print("\n"); while($server = <$machinelist>){ chomp($server); if(-d "./$server"){ copydata(); }else{ mkdir($server)||die $!; copydata(); } } sub copydata{ $scp = Net::SCP->new($server, "username"); $scp->get("/tmp/nmonout/*", "/export/nmon/$server/") or die $s +cp->{errstr}; ssh($server, "find /tmp/nmonout/ -mtime +2 |xargs rm"); }

Replies are listed 'Best First'.
Re: File copier
by salva (Abbot) on Jul 20, 2010 at 21:49 UTC
    Using Net::OpenSSH::Parallel you can run all the operations in parallel. It is probably going to be much faster:
    use Net::OpenSSH::Parallel; my $pssh = Net::OpenSSH::Parallel->new; my $machinelist; my $machinelistfilename = "./machinelist.txt"; open($machinelist, "<", $machinelistfilename) or die $!; while(<$machinelist>) { chomp; mkdir $_; -d $_ or die $!; $pssh->add_host($_, user => "username"); } $pssh->push('*', scp_get => { glob => 1 }, '/tmp/nmonout/*', "./%HOST% +/); $pssh->push('*', command => "find /tmp/nmonout/ -mtime +2 |xargs rm"); $pssh->run;
Re: File copier
by sierpinski (Chaplain) on Jul 22, 2010 at 04:43 UTC
    I did the same thing with Net::SSH::Expect and Parallel::ForkManager and it worked really well. I have one server connect to all the rest, run commands, capture the output and generate a report based on what it finds.

    If you're interested, I can post some of my code tomorrow. (Can't access it from home at the moment.)

      Please do. I always like to see how other people do the same things. Thanks

        Sorry it took longer than I thought... but here goes... I've omitted non-related portions of code, but you should get the idea:
        ($serverlist is a list of servers, 1 per line)
        my $max_procs = 20; # Create the object that will allow parallel processing my $pm = new Parallel::ForkManager($max_procs); # I used this as a counter to keep track of how many servers # have been connected to vs how many have been attempted, # and used the former as a progress indicator that I've # omitted here. $pm->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; $connected_server_count += $exit_code; $progress = ($count / $total_server_count) * 100; printf "Progress: %4.1f\%\r",$progress; $count++; } ); foreach my $host (0 .. $#serverlist) { my $pid = $pm->start($serverlist[$host]) and next; # Create ssh object to one server my $ssh = Net::SSH::Expect->new ( host => "$serverlist[$host]", user => "$user", raw_pty => 1, restart_timeout_upon_receive => 1, timeout => 6, ssh_option => " -x", log_file=> "/tmp/$serverlist[$host].log", ); # Validate that we have a successful ssh connection $login_output = $ssh->run_ssh() or die "Could start ssh proces +s, error $!\n"; sleep(1); my $ret; my $rc1 = eval{$ret = $ssh->read_all(12);}; unless($rc1) { open(NOCON,">>$NOCONNECT"); print NOCON "$serverlist[$host] - Could not connect -- + skipping.\n"; close(NOCON); $pm->finish(0); # Couldn't connect to this one, skip +to next server in main loop } my $rc = eval{( $ret =~ />\s*|$\s*\z/) or die "where's the rem +ote prompt?";}; if($rc) { if($ret =~ m/[Pp]assword:/) { # print("Server asking for password, key not ins +talled.\n"); open(NOCON,">>$NOCONNECT"); print NOCON "$serverlist[$host] - Asking for p +assword -- skipping.\n"; close(NOCON); $pm->finish(0); # Couldn't connect to this +one, skip to next server in main loop } } $ssh->exec("stty raw -echo"); # Perform all of your checks here, I'll add one for example # Execute dumpadm check - solaris coredump location my $dumpadm_output = $ssh->send("$dumpadm_cmd"); my $line; while ( defined ($line = $ssh->read_line()) ) { if($line =~ m/dedicated/) { if($line =~ m/zvol/) { # do nothing, uses zfs print("Dumpadm uses zvol -- rootfs is +zfs.\n"); } else { open(DUMP,">>$DUMPADM"); print DUMP "$serverlist[$host] $line \ +n"; close(DUMP); } } } # Close ssh connection to this server $ssh->close(); $pm->finish(1); # 1 is successful, returned 0 for failure to +connect } # End of main while loop for server list # Since multiple processes were spawned asynchronously, must wait unti +l they're all finished # before continuing. $pm->wait_all_children; # won't exit completely until all children are + finished # At this point, I just took all my temporary files and cat'd them tog +ether and sent it in an email to my team
        Since I just copied/pasted chunks of code, I doubt it would work if you tried to run it (even after setting the necessary variables) but you should get the idea of how it's set up. Hope that helps!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://850494]
Approved by ww
Front-paged by Arunbear
and a soft breeze sighs...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (3)
As of 2017-07-24 07:34 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (348 votes). Check out past polls.