SSH2 - Asynchronous Opens & Synchronous Commands

[5haun]

Gracious Monks, I seek your counsel.

I have been tasked with integrating SSH2 support into an existing in-house framework, and I have successfully done so using the wonderful Control::CLI module, which I have enhanced with some additional functionality. However, one of the requirements is for the initial connections to the devices to be done asynchronously, as that is the slowest part of the process. So far, I can only thread the entire communication with a device (the connection, actions, reactions, and close for a given device all within one subroutine).

Basically, the request is to have the connect() calls performed simultaneously for a given array of devices and then have the rest of the operations performed serially.

For example, connections are simultaneously created to devices A, B, and C. After waiting for all the devices to be connected, an action (which is an interaction with the remote shell) is performed on device A, which could potentially affect the state on devices B or C, so the state on those devices are then inspected. This may result in an action being performed on B or C, which in turn would require the states on the other devices to be re-inspected. If communication is lost with a device, it would need to be re-established before proceeding.

A similar solution has already been done using ithreads with Telnet. The opens are performed in child threads, but the rest of the Telnet commands occur in thread 0. I don't think I will be able to use that solution with SSH, as I'd need a way of sharing the Control::CLI (i.e. Net::SSH2) object instead of having it copied to the children (threads::shared does not appear to support the Net::SSH2/Control::CLI object). I've also read here that it may be an SSH security violation to have the parent and its child thread both access the same session. If someone knows a way to do this with ithreads, I'd love to understand how it can be done. :)

Other options under consideration are to use an event loop (POE or IO::Async) or fork a process for each device and have the parent use some sort of message queue to communicate with each child (I'd need STDIN, STDOUT, STDERR, and return codes to be communicated). Right now my best option appears to figure out a way to leverage POE::Component::Generic::Net::SSH2.

Thanks in advance for your time, wisdom, and constructive suggestions.

[BrowserUk]

I'd need a way of sharing the Control::CLI (i.e. Net::SSH2) object instead of having it copied to the children (threads::shared does not appear to support the Net::SSH2/Control::CLI object). I've also read here that it may be an SSH security violation to have the parent and its child thread both access the same session. If someone knows a way to do this with ithreads, I'd love to understand how it can be done. :)

Try:

use threads;
use threads::shared qw[ shared_clone ];

my @sessionData = ...;

my @ssh2_handles :shared;

sub opener {
    my( $idx, ... ) = @_;
    use Net::SSH2;
    
    my $ssh = Net::SSH2->new( .. );
    lock @ssh2_handles;
    $ssh2_handles[ $idx ] = shared_clone( $ssh );
    return;
}

my $idx = 0;
my @openers = async( \&opener, $idx++, $_ ) for @sessionData;

$_->join for @openers;

### use @ssh2_handles here...

This is untested, untried, speculation -- I don't have facilities to do so -- but the basic premise is sound.

[5haun]

Your approach to the threads issue is worth a try. I'll test and report back. Thanks.

[5haun]

Well, I can't get passed the segfault when connect() is called. I created an unthreaded Net::SSH2 test to ensure it works unthreaded, then applied your suggestions. Here is the result:

$ ./threaded-test7.pl 10.88.88.88 username password 
num hosts: 1
num openers created: 1
0, 10.88.88.88
openers completed
dump of ssh2_handle:
bless(do{\(my $o = 140010028283744)}, "Net::SSH2")
dump of $ssh2_handles[0]:
bless(do{\(my $o = 140010028283744)}, "Net::SSH2")
Trying to connect to 10.88.88.88
Segmentation fault (core dumped)

Here is the code:

#!/usr/bin/env perl

use threads;
use threads::shared qw[ shared_clone ];
use v5.14.2;
use strict;
use Data::Dump qw(dd pp);

my (@hosts) = split /,/, shift @ARGV || usage('Missing host[,host2,...
+]');
say "num hosts: " . scalar(@hosts);
my $user = shift(@ARGV) || usage('Missing user');
my $pass = shift(@ARGV) || usage('Missing password');
my $port = 22;
my @ssh2_handles : shared;

sub opener {
    use Net::SSH2;
    my ( $idx, $host ) = @_;
    say "$idx, $host";
    my $ssh2 = Net::SSH2->new( trace => -1 );
    lock @ssh2_handles;
    $ssh2_handles[$idx] = shared_clone($ssh2);
    return;
}

my $idx = 0;
# my @openers = async( \&opener, $idx++, $_ ) for @hosts;  # this crea
+tes zero elements
my @openers;
push @openers, async( \&opener, $idx++, $_ ) for @hosts;
say "num openers created: " . scalar(@openers);
$_->join for @openers;
say "openers completed";

my $host        = $hosts[0];
my $ssh2_handle = $ssh2_handles[0];
say "dump of ssh2_handle:";
dd $ssh2_handle;
say 'dump of $ssh2_handles[0]:';
dd $ssh2_handles[0];

my $ok;
eval {
    say "Trying to connect to $host";
    $ok = $ssh2_handle->connect( $host, $port );
};
die "Whoops: $@" if $@;
die "SSH unable to connect for some reason" unless $ok;
say "Connected to '$host'";

$ssh2_handle->auth_password( $user, $pass ) // $ssh2_handle->auth_pass
+word( $user, $pass )
    // die "ERROR: Failed to authenticate";
say "Authenticated as user '$user'";

sub usage {
    my ($msg) = @_;
    say STDERR "FATAL: $msg" if $msg;
    say STDERR "Usage: $0 host[,host2,...] user password";
    exit(1);
}

Any more ideas or suggestions? Thanks!

Here is the unthreaded result:

$ ./unthreaded-test7.pl 10.88.88.88 admin password                    
+        
num hosts: 1
0, 10.88.88.88
dump of ssh2_handle:
bless(do{\(my $o = 39570368)}, "Net::SSH2")
dump of $ssh2_handles[0]:
bless(do{\(my $o = 39570368)}, "Net::SSH2")
Trying to connect to 10.88.88.88
Connected to '10.88.88.88'
Authenticated as user 'admin'

and code:

#!/usr/bin/env perl

use threads;
use threads::shared qw[ shared_clone ];
use v5.14.2;
use strict;
use Data::Dump qw(dd pp);

my (@hosts) = split /,/, shift @ARGV || usage('Missing host[,host2,...
+]');
say "num hosts: " . scalar(@hosts);
my $user = shift(@ARGV) || usage('Missing user');
my $pass = shift(@ARGV) || usage('Missing password');
my $port = 22;
my @ssh2_handles : shared;

use Net::SSH2;
my ( $idx, $host ) = (0,$hosts[0]);
say "$idx, $host";
my $ssh2 = Net::SSH2->new( trace => -1 );

my $host        = $hosts[0];
my $ssh2_handle = $ssh2;
say "dump of ssh2_handle:";
dd $ssh2_handle;
say 'dump of $ssh2_handles[0]:';
dd $ssh2;

my $ok;
eval {
    say "Trying to connect to $host";
    $ok = $ssh2_handle->connect( $host, $port );
};
die "Whoops: $@" if $@;
die "SSH unable to connect for some reason" unless $ok;
say "Connected to '$host'";

$ssh2_handle->auth_password( $user, $pass ) // $ssh2_handle->auth_pass
+word( $user, $pass )
    // die "ERROR: Failed to authenticate";
say "Authenticated as user '$user'";

sub usage {
    my ($msg) = @_;
    say STDERR "FATAL: $msg" if $msg;
    say STDERR "Usage: $0 host[,host2,...] user password";
    exit(1);
}

[5haun]

BTW, I tried moving the connect inside of opener(). The open now succeeds, but the authentication now segfaults. There is something wrong with using shared handle outside the thread. Thanks.

[BrowserUk]

BTW, I tried moving the connect inside of opener(). The open now succeeds, but the authentication now segfaults.

What you've conclusively proved is that either Net::SSH2, or more likely, the underlying C libraries it uses, are not reentrant. Ie. Not threadsafe.

There is something wrong with using shared handle outside the thread.

Nope. The main thread of a process is a thread just like any other threads you create.

You will just have to accept that there is nothing you can do to fix or work around the problems of the module and its underlying libraries.

[salva]

Check Net::OpenSSH and Net::OpenSSH::Parallel.

[5haun]

Thanks for the suggestion Salva, but I don't believe your modules will suffice in this case. Net::OpenSSH lacks sufficient shell support for this use case, and Net::OpenSSH::Parallel is does not fit the use case (only the connect() calls are to be done in parallel - I can't queue up a bunch of commands and run).

I know you strongly believe the idea of having to interact with the shell is a bad one, but that is the requirement. The framework we've built is used to verify command sets executed in the remote shell environment of hosts and devices. I can't just connect, run a command, and close. I need to connect, send a remote command (which could be a shell builtin like cd), check the status, run a command, inspect the output, and so on. It is an automated interaction. Combining commands with '&' isn't an option - they need to be executed individually and evaluated. Basically, I need the full functionality of Net::SSH::Expect, which we've also implemented, but this task to implement Control::CLI/Net::SSH2 is to replace the use of Net::SSH::Expect, as Net::SSH::Expect is not thread-safe.

If Net::OpenSSH had shell support like Net::SSH2, I'd try it again in a heartbeat.

Net::SSH2 is also thread-safe. At least it is when the entire test for a device is contained within a thread. The questing being asked is what is the best method of making only the connections in parallel (for performance reasons) and still allow the remainder of the test to be performed serially?

[salva]

You can run shells with Net::OpenSSH quite easily and even use Expect to interact with them:

my %ssh;
for my $host (@hosts) {
  $ssh{$host} = Net::OpenSSH->new($host, async => 1, ...);
}

for my $host (@hosts) {
  my $ssh = $ssh{$host};
  my ($pty, $pid) = $ssh->open2pty; # whith out command launchs a shel
+l
  my $expect = Expect->init($pty);
  $expect->...

}

[Theodore]

The parent/childs solution seems elegant and interesting. A child for each device, opening the ssh connection and then receiving instructions and sending results through an IPC message queue. You could parse the commands' output in the childs' code and use a simple protocol to communicate with the parent.

[5haun]

We may end up having to do something like that, but the requirement is for the output parsing to be done by the caller:

caller test script => framework library => ssh module => remote shell => remote command => output => ssh module => framework library => caller test script

Thanks for the input.

[zentara]

As I tried to understand what your question really involved, it dawned on me, that maybe using an SSH client to do this, is doing it the hard way. Why not open your own direct socket connections between machines? You can also make your direct socket connections very secure using something like Net::EasyTCP.

See Glib based forking server with root messaging for a Glib eventloop based socket network. See Tk encrypted echoing-chat client and server for an example of encrypted connections.

[salva]

libssh2 can do the SSH handshaking and user authentication stages in non-blocking mode. I am not completely sure the same can be currently done with Net::SSH2 though.

The trick would be to call $ssh2->blocking(0) just after creating the Net::SSH2 object and then calling connect and auth_* while checking for LIBSSH2_ERROR_EAGAIN errors and then using the select function to wait for data to arrive at some of the underlaying sockets.

I have done something similar for the Net::SSH2 backend of my module Net::SSH::Any and would have to say that it is going to be a difficult task and would probably require fixing bugs in Net::SSH2 (in that case, report them through GitHub, I would be looking and could help).

[5haun]

Hhmmm. Ok. I'll give it a try...

[stonecolddevin]

I only glossed over this, but if you're doing the same thing on several devices, try pssh. It's like a super duper boiled down Chef/Puppet that allows you to execute remote calls over ssh in parallel.

[sundialsvc4]

I perceive that you will probably need to spawn a child process for each device and to use a message queue.   The real difficulty is going to be how to describe the state-machine type logic that must be performed by the main process that is controlling the tests.   I vaguely remember encountering such a testing framework – I no longer remember what its name is or was – but it was built using XML, vaguely like what ColdFusion does.

The test file consisted of an arbitrary number of test-nodes, one being designated as the initial-state.   Each one had “preconditions,” “actions,” and “effects.”   They could be “singleton,” or “parallel.”     The test-runner used this information to decide what to tell its slaves to do next.   I recall that while the framework worked, it was .. difficult .. to program tests for it . . .

Thinking a little bit more about this, I do believe that it is the right approach to have n processes, one per SSH2 connection, who are basically “proxies” for that connection and who are directly responsible for (a) knowing what that host is or isn’t doing at this time, and (b) telling the host to do whatever-it-is and to correctly report that host’s new status.   They are, effectively, “supervisors” for each host, and the first line of defense in dealing with hiccups.   Then, the remaining process-of-significance (probably, the last remaining child of the parent) is the test-runner process itself, which acts by sending and receiving messages via IPC queues to each host-supervisor.