Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

SSH2 - Asynchronous Opens & Synchronous Commands

by 5haun (Sexton)
on Apr 04, 2014 at 09:53 UTC ( #1081100=perlquestion: print w/ replies, xml ) Need Help??
5haun has asked for the wisdom of the Perl Monks concerning the following question:

Gracious Monks, I seek your counsel.

I have been tasked with integrating SSH2 support into an existing in-house framework, and I have successfully done so using the wonderful Control::CLI module, which I have enhanced with some additional functionality. However, one of the requirements is for the initial connections to the devices to be done asynchronously, as that is the slowest part of the process. So far, I can only thread the entire communication with a device (the connection, actions, reactions, and close for a given device all within one subroutine).

Basically, the request is to have the connect() calls performed simultaneously for a given array of devices and then have the rest of the operations performed serially.

For example, connections are simultaneously created to devices A, B, and C. After waiting for all the devices to be connected, an action (which is an interaction with the remote shell) is performed on device A, which could potentially affect the state on devices B or C, so the state on those devices are then inspected. This may result in an action being performed on B or C, which in turn would require the states on the other devices to be re-inspected. If communication is lost with a device, it would need to be re-established before proceeding.

A similar solution has already been done using ithreads with Telnet. The opens are performed in child threads, but the rest of the Telnet commands occur in thread 0. I don't think I will be able to use that solution with SSH, as I'd need a way of sharing the Control::CLI (i.e. Net::SSH2) object instead of having it copied to the children (threads::shared does not appear to support the Net::SSH2/Control::CLI object). I've also read here that it may be an SSH security violation to have the parent and its child thread both access the same session. If someone knows a way to do this with ithreads, I'd love to understand how it can be done. :)

Other options under consideration are to use an event loop (POE or IO::Async) or fork a process for each device and have the parent use some sort of message queue to communicate with each child (I'd need STDIN, STDOUT, STDERR, and return codes to be communicated). Right now my best option appears to figure out a way to leverage POE::Component::Generic::Net::SSH2.

Thanks in advance for your time, wisdom, and constructive suggestions.

Comment on SSH2 - Asynchronous Opens & Synchronous Commands
Re: SSH2 - Asynchronous Opens & Synchronous Commands
by salva (Monsignor) on Apr 04, 2014 at 10:22 UTC

      Thanks for the suggestion Salva, but I don't believe your modules will suffice in this case. Net::OpenSSH lacks sufficient shell support for this use case, and Net::OpenSSH::Parallel is does not fit the use case (only the connect() calls are to be done in parallel - I can't queue up a bunch of commands and run).

      I know you strongly believe the idea of having to interact with the shell is a bad one, but that is the requirement. The framework we've built is used to verify command sets executed in the remote shell environment of hosts and devices. I can't just connect, run a command, and close. I need to connect, send a remote command (which could be a shell builtin like cd), check the status, run a command, inspect the output, and so on. It is an automated interaction. Combining commands with '&' isn't an option - they need to be executed individually and evaluated. Basically, I need the full functionality of Net::SSH::Expect, which we've also implemented, but this task to implement Control::CLI/Net::SSH2 is to replace the use of Net::SSH::Expect, as Net::SSH::Expect is not thread-safe.

      If Net::OpenSSH had shell support like Net::SSH2, I'd try it again in a heartbeat.

      Net::SSH2 is also thread-safe. At least it is when the entire test for a device is contained within a thread. The questing being asked is what is the best method of making only the connections in parallel (for performance reasons) and still allow the remainder of the test to be performed serially?

        You can run shells with Net::OpenSSH quite easily and even use Expect to interact with them:
        my %ssh; for my $host (@hosts) { $ssh{$host} = Net::OpenSSH->new($host, async => 1, ...); } for my $host (@hosts) { my $ssh = $ssh{$host}; my ($pty, $pid) = $ssh->open2pty; # whith out command launchs a shel +l my $expect = Expect->init($pty); $expect->... }
Re: SSH2 - Asynchronous Opens & Synchronous Commands
by Theodore (Scribe) on Apr 04, 2014 at 11:30 UTC
    The parent/childs solution seems elegant and interesting. A child for each device, opening the ssh connection and then receiving instructions and sending results through an IPC message queue. You could parse the commands' output in the childs' code and use a simple protocol to communicate with the parent.

      We may end up having to do something like that, but the requirement is for the output parsing to be done by the caller:

      caller test script => framework library => ssh module => remote shell => remote command => output => ssh module => framework library => caller test script

      Thanks for the input.

Re: SSH2 - Asynchronous Opens & Synchronous Commands
by BrowserUk (Pope) on Apr 04, 2014 at 11:57 UTC
    I'd need a way of sharing the Control::CLI (i.e. Net::SSH2) object instead of having it copied to the children (threads::shared does not appear to support the Net::SSH2/Control::CLI object). I've also read here that it may be an SSH security violation to have the parent and its child thread both access the same session. If someone knows a way to do this with ithreads, I'd love to understand how it can be done. :)

    Try:

    use threads; use threads::shared qw[ shared_clone ]; my @sessionData = ...; my @ssh2_handles :shared; sub opener { my( $idx, ... ) = @_; use Net::SSH2; my $ssh = Net::SSH2->new( .. ); lock @ssh2_handles; $ssh2_handles[ $idx ] = shared_clone( $ssh ); return; } my $idx = 0; my @openers = async( \&opener, $idx++, $_ ) for @sessionData; $_->join for @openers; ### use @ssh2_handles here...

    This is untested, untried, speculation -- I don't have facilities to do so -- but the basic premise is sound.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Your approach to the threads issue is worth a try. I'll test and report back. Thanks.

      Well, I can't get passed the segfault when connect() is called. I created an unthreaded Net::SSH2 test to ensure it works unthreaded, then applied your suggestions. Here is the result:

      $ ./threaded-test7.pl 10.88.88.88 username password num hosts: 1 num openers created: 1 0, 10.88.88.88 openers completed dump of ssh2_handle: bless(do{\(my $o = 140010028283744)}, "Net::SSH2") dump of $ssh2_handles[0]: bless(do{\(my $o = 140010028283744)}, "Net::SSH2") Trying to connect to 10.88.88.88 Segmentation fault (core dumped)

      Here is the code:

      #!/usr/bin/env perl use threads; use threads::shared qw[ shared_clone ]; use v5.14.2; use strict; use Data::Dump qw(dd pp); my (@hosts) = split /,/, shift @ARGV || usage('Missing host[,host2,... +]'); say "num hosts: " . scalar(@hosts); my $user = shift(@ARGV) || usage('Missing user'); my $pass = shift(@ARGV) || usage('Missing password'); my $port = 22; my @ssh2_handles : shared; sub opener { use Net::SSH2; my ( $idx, $host ) = @_; say "$idx, $host"; my $ssh2 = Net::SSH2->new( trace => -1 ); lock @ssh2_handles; $ssh2_handles[$idx] = shared_clone($ssh2); return; } my $idx = 0; # my @openers = async( \&opener, $idx++, $_ ) for @hosts; # this crea +tes zero elements my @openers; push @openers, async( \&opener, $idx++, $_ ) for @hosts; say "num openers created: " . scalar(@openers); $_->join for @openers; say "openers completed"; my $host = $hosts[0]; my $ssh2_handle = $ssh2_handles[0]; say "dump of ssh2_handle:"; dd $ssh2_handle; say 'dump of $ssh2_handles[0]:'; dd $ssh2_handles[0]; my $ok; eval { say "Trying to connect to $host"; $ok = $ssh2_handle->connect( $host, $port ); }; die "Whoops: $@" if $@; die "SSH unable to connect for some reason" unless $ok; say "Connected to '$host'"; $ssh2_handle->auth_password( $user, $pass ) // $ssh2_handle->auth_pass +word( $user, $pass ) // die "ERROR: Failed to authenticate"; say "Authenticated as user '$user'"; sub usage { my ($msg) = @_; say STDERR "FATAL: $msg" if $msg; say STDERR "Usage: $0 host[,host2,...] user password"; exit(1); }

      Any more ideas or suggestions? Thanks!

      Here is the unthreaded result:

      $ ./unthreaded-test7.pl 10.88.88.88 admin password + num hosts: 1 0, 10.88.88.88 dump of ssh2_handle: bless(do{\(my $o = 39570368)}, "Net::SSH2") dump of $ssh2_handles[0]: bless(do{\(my $o = 39570368)}, "Net::SSH2") Trying to connect to 10.88.88.88 Connected to '10.88.88.88' Authenticated as user 'admin'

      and code:

      #!/usr/bin/env perl use threads; use threads::shared qw[ shared_clone ]; use v5.14.2; use strict; use Data::Dump qw(dd pp); my (@hosts) = split /,/, shift @ARGV || usage('Missing host[,host2,... +]'); say "num hosts: " . scalar(@hosts); my $user = shift(@ARGV) || usage('Missing user'); my $pass = shift(@ARGV) || usage('Missing password'); my $port = 22; my @ssh2_handles : shared; use Net::SSH2; my ( $idx, $host ) = (0,$hosts[0]); say "$idx, $host"; my $ssh2 = Net::SSH2->new( trace => -1 ); my $host = $hosts[0]; my $ssh2_handle = $ssh2; say "dump of ssh2_handle:"; dd $ssh2_handle; say 'dump of $ssh2_handles[0]:'; dd $ssh2; my $ok; eval { say "Trying to connect to $host"; $ok = $ssh2_handle->connect( $host, $port ); }; die "Whoops: $@" if $@; die "SSH unable to connect for some reason" unless $ok; say "Connected to '$host'"; $ssh2_handle->auth_password( $user, $pass ) // $ssh2_handle->auth_pass +word( $user, $pass ) // die "ERROR: Failed to authenticate"; say "Authenticated as user '$user'"; sub usage { my ($msg) = @_; say STDERR "FATAL: $msg" if $msg; say STDERR "Usage: $0 host[,host2,...] user password"; exit(1); }
      BTW, I tried moving the connect inside of opener(). The open now succeeds, but the authentication now segfaults. There is something wrong with using shared handle outside the thread. Thanks.
        BTW, I tried moving the connect inside of opener(). The open now succeeds, but the authentication now segfaults.

        What you've conclusively proved is that either Net::SSH2, or more likely, the underlying C libraries it uses, are not reentrant. Ie. Not threadsafe.

        There is something wrong with using shared handle outside the thread.

        Nope. The main thread of a process is a thread just like any other threads you create.

        You will just have to accept that there is nothing you can do to fix or work around the problems of the module and its underlying libraries.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: SSH2 - Asynchronous Opens & Synchronous Commands
by sundialsvc4 (Abbot) on Apr 04, 2014 at 13:31 UTC

    I perceive that you will probably need to spawn a child process for each device and to use a message queue.   The real difficulty is going to be how to describe the state-machine type logic that must be performed by the main process that is controlling the tests.   I vaguely remember encountering such a testing framework I no longer remember what its name is or was but it was built using XML, vaguely like what ColdFusion does.

    The test file consisted of an arbitrary number of test-nodes, one being designated as the initial-state.   Each one had “preconditions,” “actions,” and “effects.”   They could be “singleton,” or “parallel.”     The test-runner used this information to decide what to tell its slaves to do next.   I recall that while the framework worked, it was .. difficult .. to program tests for it . . .

    Thinking a little bit more about this, I do believe that it is the right approach to have n processes, one per SSH2 connection, who are basically “proxies” for that connection and who are directly responsible for (a) knowing what that host is or isn’t doing at this time, and (b) telling the host to do whatever-it-is and to correctly report that host’s new status.   They are, effectively, “supervisors” for each host, and the first line of defense in dealing with hiccups.   Then, the remaining process-of-significance (probably, the last remaining child of the parent) is the test-runner process itself, which acts by sending and receiving messages via IPC queues to each host-supervisor.

Re: SSH2 - Asynchronous Opens & Synchronous Commands
by zentara (Archbishop) on Apr 04, 2014 at 14:22 UTC
Re: SSH2 - Asynchronous Opens & Synchronous Commands
by stonecolddevin (Vicar) on Apr 04, 2014 at 17:53 UTC

    I only glossed over this, but if you're doing the same thing on several devices, try pssh. It's like a super duper boiled down Chef/Puppet that allows you to execute remote calls over ssh in parallel.

    Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

Re: SSH2 - Asynchronous Opens & Synchronous Commands
by salva (Monsignor) on Apr 05, 2014 at 08:40 UTC
    libssh2 can do the SSH handshaking and user authentication stages in non-blocking mode. I am not completely sure the same can be currently done with Net::SSH2 though.

    The trick would be to call $ssh2->blocking(0) just after creating the Net::SSH2 object and then calling connect and auth_* while checking for LIBSSH2_ERROR_EAGAIN errors and then using the select function to wait for data to arrive at some of the underlaying sockets.

    I have done something similar for the Net::SSH2 backend of my module Net::SSH::Any and would have to say that it is going to be a difficult task and would probably require fixing bugs in Net::SSH2 (in that case, report them through GitHub, I would be looking and could help).

      Hhmmm. Ok. I'll give it a try...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1081100]
Approved by Bloodnok
Front-paged by Bloodnok
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (12)
As of 2014-10-31 23:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (225 votes), past polls