Re: Net::OpenSSH reconnect master if conn. closed

[haukex]

I'm not sure at the moment if there's a built-in way to do this. But here is a stripped-down version of a script I've had in active use for about a year now to maintain a connection to an SSH server:

use warnings;
use strict;
use Net::OpenSSH;

my $connect_interval_s = 60;
my $command_interval_s = 60;

my $run=1;
local $SIG{INT}  = sub { $run=0 };
local $SIG{TERM} = sub { $run=0 };
OUTER: while ($run) {
    my $ssh;
    eval {
        $ssh = Net::OpenSSH->new( 'hostname', user => 'username',
                timeout => 10, kill_ssh_on_timeout => 1,
            ) or die "Failed to set up SSH";
        die "SSH failed to connect: ".$ssh->error if $ssh->error;
    ;1} or do {
        warn "Connect failed, will retry: $@";
        sleep $connect_interval_s if $run;
        redo OUTER;
    };
    # connection is up
    INNER: while ($run) {
        eval {
            my $rv = $ssh->capture('echo some command');
            die "Remote command failed: ".$ssh->error if $ssh->error;
            # ... do something with $rv here ...
        ;1} or do {
            warn "Command failed, will attempt to reconnect: $@";
            last INNER;
        };
        sleep $command_interval_s if $run;
    }
    sleep $connect_interval_s if $run;
}

In addition, I've got this in my ~/.ssh/config for that hostname:

Host hostname
    User username
    IdentityFile ~/.ssh/filename
    ConnectTimeout 10
    ServerAliveInterval 10
    ServerAliveCountMax 3
    ExitOnForwardFailure yes

[Bolemo]

Interesting! Thank you.

I will keep that in mind. I will first experiment with a simple $ssh->check_master

Let’s say that I created a master connection this way: my $ssh = Net::OpenSSH->new(...);

Before calling a slave ssh command with open3 or capture, I will test $ssh to check if it is still connected or not, like:

$ssh = Net::OpenSSH->new(...) unless ($ssh->check_master);
$ssh->capture(...);


I will see how light or heavy on CPU this check_master function is.
EDIT: no errors and the script works fine. Killed the ssh process on the remote device to test and...
The script works, but not as expected. Seems that an ssh connection is recreated allowing the slave command to be run, but there is no more master ssh connection sticking. After first disconnect event, it reconnects each time instead of staying connected like the first time.

[salva]

I have added a new restart method for reestablishing the connection to the server in Net::OpenSSH version 0.80. It is something that should have been available a long time ago, but at first it was difficult to implement, then at some point I had to revamp the code managing the master process in order to fix several others bugs making it trivial but didn't though about adding it... until I read your post today!

Note that it is possible that a broken connection goes unnoticed until you actually try to use it. wait_for_master cheeks the local ssh master process but some kind of errors (for instance, a broken TCP connection), would go unnoticed. So to be sure you cover all the cases, you need to try running the command and if that fails, reconnect:

my $cmd = "...";

my $out = $ssh->capture({timeout => 120}, $cmd);
unless (defined $out) {
  warn "unable to run command $cmd, retrying...";
  $ssh->restart or die "Unable to restart SSH connection: " . $ssh->er
+ror;
  $out = $ssh->capture({timeout => 120}, $cmd);
  unless (defined $out) {
    die "definitively, I can't run the command $cmd: " . $ssh->error;
  }
}

If $cmd does some non-atomic operation, it is a long running operation you don't want to run twice, etc., you may like to use a harmless dummy command as "echo hello" instead.

[Bolemo]

That is great, thank you!