Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

RFC and Questions regarding a module to automate and centralize infrastructure administration

by cbeckley (Curate)
on Mar 24, 2017 at 23:57 UTC ( #1185850=perlquestion: print w/replies, xml ) Need Help??

cbeckley has asked for the wisdom of the Perl Monks concerning the following question:

I've written a module, nascent, that makes it a little easier to execute a list of commands on a group of servers. Basically I create an array of servers, which I refer to as contexts in the code, each of which has an array of commands and all the commands get executed in all the contexts and all the output is gathered and returned for processing.

I've included below an excerpt from an invoking script to give an example of how the data structure is built, and below that, the module itself.

First, a couple questions.

Has this been done before? I mean, it has to have, right? My searches haven't turned up much however. nysus seems to be working on something similar, Re: How best to validate the keys of hashref arguments?, and melezhik recently wrote about Sparrow - your own script manager, which seems to be in the neighborhood, but not quite what I'm looking for. If anybody knows of a module that already does this, I'd really appreciate it.

How do I organize my modules? Assuming I do continue this development, I plan on rewriting contexts and commands as objects, which would by my first foray into objects with Perl, and I'm not sure how they should exist with respect to each other. In general, what are the implications between Foo::Bar and Foo::Baz as opposed to Foo::Bar and Foo::Bar::Baz? And in this case, Ops::Context and Ops::Command, where Ops::Context would use Ops::Command and a calling script would have to use both, or Ops::Command and Ops::Command::Context where Contexts are extensions of Commands and the invoking script only uses one or the other, depending on the complexity of the task?

Finally, any feedback on the code below would be appreciatel, any smoking guns that would explain this hole in my foot, or anything I may have done clumsily for which Perl has a better construct.

For instance, the line where I access the array of commands:

for my $cmd (@{$ctxt->{commands}}) { ...
Is there a better way to say that?

An excerpt from a script that invokes the module:

use Ops::Exec qw(ops_do_ssh_shell ops_new_context ops_new_cmd); use Ops::Config qw(ops_get_servers); my %servers = map { $_ => new_server($_) } ops_get_servers('foo'); my @contexts = map {ops_new_context({ name => $servers{$_}{name}, user => 'foo', host => $servers{$_}{host}, key => $ENV{HOME} . '/.ssh/id_rsa.foo.prod', commands => [ ops_new_cmd({ name => 'capacity', command => 'df -k /home'}), ops_new_cmd({ name => 'process_type1', command => 'ps -ef | grep process_ty +pe1 | grep -v grep | wc -l'}), ops_new_cmd({ name => 'process_type2', command => 'ps -ef | grep process_ty +pe2 | grep -v grep | wc -l'}) ] })} sort keys %servers; for my $ctxt (@contexts) { if ($local_state->{status} eq 'success') { my $ctxt_status = ops_do_ssh_shell($ctxt); for my $cmd_status (@{$ctxt_status->{commands}}) { if ( $ctxt_status->{success} ) { $servers{$ctxt_status->{name}}{$cmd_status->{name}} = $cmd +_status->{output}; } else { $local_state->{status} = 'failure'; $local_state->{stack_trace} = $cmd_status; } } } } if ($local_state->{status} eq 'success') { for my $server (keys %servers) { $servers{$server}{capacity} = parse_capacity($servers{$server}{c +apacity}); # further parsing and validation } } # statuses are evaluated to generate warnings and alerts # texts and emails are sent yadda yadda

And the module:

package Ops::Exec; use strict; use warnings; use Exporter qw(import); use Data::Dumper; use Net::OpenSSH; use Hash::Util qw(lock_ref_keys); use Clone qw(clone); our @EXPORT_OK = qw(ops_do_ssh_shell ops_do_ssh_qx ops_new_context ops +_new_cmd ops_new_qx_cmd); use constant SUCCESS => 1; use constant FAILURE => 0; sub ops_new_cmd { my ($init_hash) = @_; my $new_cmd = {}; lock_ref_keys($new_cmd, qw(name command success output std_err cmd_ +ret_code cmd_ret_msg)); $new_cmd = clone($init_hash); $new_cmd->{success} = SUCCESS; return $new_cmd } sub ops_new_context { my ($init_hash) = @_; my $new_cmd = {}; lock_ref_keys($new_cmd, qw(name user host key commands success ssh_ +retcode ssh_retmsg outputs)); $new_cmd = clone($init_hash); $new_cmd->{success} = SUCCESS; return $new_cmd } sub ops_do_ssh_shell { my ($ctxt) = @_; my $ssh; defined $ctxt->{key} or $ctxt->{key} = $ENV{HOME} . '/.ssh/id_rsa'; if ( $ctxt->{success} ) { $ssh = Net::OpenSSH->new($ctxt->{host}, user => $ctxt->{user}, k +ey_path => $ctxt->{key}); if ( $ssh->error ) { $ctxt->{success} = FAILURE; ($ctxt->{ssh_retcode}, $ctxt->{ssh_retmsg}) = (0 + $ssh->erro +r, '' . $ssh->error); } } for my $cmd (@{$ctxt->{commands}}) { if ( $ctxt->{success} ) { ($cmd->{output}, $cmd->{std_err}) = $ssh->capture2($cmd->{com +mand}); $cmd->{cmd_ret_code} = 0 + $ssh->error; if ( $cmd->{cmd_ret_code} ) { $ctxt->{success} = FAILURE; $ctxt->{cmd_ret_msg} = '' . $ssh->error; } chomp $cmd->{output}; chomp $cmd->{std_err}; } } return $ctxt; } sub ops_new_qx_cmd { my ($init_hash) = @_; my $new_cmd = {}; lock_ref_keys($new_cmd, qw(name user host command success output st +d_err cmd_ret_code cmd_ret_msg ssh_ret_code ssh_ret_msg ssh_cmd_qx)); %$new_cmd = %$init_hash; $new_cmd->{success} = SUCCESS; return $new_cmd } sub ops_do_ssh_qx { my ($cmd) = @_; $cmd->{ssh_cmd_qx} = 'ssh ' . $cmd->{user} . '\@' . $cmd->{host} . + ' \'' . $cmd->{command} . '\'' . ' 2>/dev/null'; $cmd->{output} = qx($cmd->{ssh_cmd_qx}); if ( defined $cmd->{output} ) { $cmd->{cmd_ret_code} = $?; chomp $cmd->{output}; } else { ($cmd->{ssh_ret_code}, $cmd->{ssh_ret_msg}) = (0 + $!, '' . $!); } return $cmd; } "Red Guitar, Three Chords ...";

Thank you for taking the time to look at this.


Thanks,
cbeckley

  • Comment on RFC and Questions regarding a module to automate and centralize infrastructure administration
  • Select or Download Code

Replies are listed 'Best First'.
Re: RFC and Questions regarding a module to automate and centralize infrastructure administration
by stevieb (Canon) on Mar 25, 2017 at 01:26 UTC
    "In general, what are the implications between Foo::Bar and Foo::Baz as opposed to Foo::Bar and Foo::Bar::Baz"

    Foo::Bar and Foo::Bar::Baz are within the same second level namespace hierarchy. That means you can have module files under the same directory structure:

    Foo-Bar | /lib | /Foo | |--Bar.pm | /Bar | |--Baz.pm

    The former, you'd be taking up two slots at the top level namespace, which typically isn't desired from the CPAN perspective unless it coincides with other work. Always try to fit what you have into an existing namespace if possible, and try to keep things contained, instead of spreading them out. For instance, what you're trying to do seems like it would fit under the Net:: top level namespace. If that's the case, a consideration would be:

    Net::Circumstance::Automate; # main module Net::Circumstance::Automate::Command; Net::Circumstance::Automate::DB; Net::Circumstance::Automate::Config; Net::Circumstance::Automate::SSH; Net::Circumstance::Automate::RDP;

    Then, in your main module, you could suck in all of the lower modules, and return new objects based on parameters to the constructor call, or have separate methods depending on what needs to be done. The config and db modules could be called early on within a constructor to set things up for any following calls.

    The dir structure would look like this:

    lib/ - Circumstance |- Automate | |- Command.pm | |- Config.pm | |- DB.pm | |- RDP.pm | |- SSH.pm | |- Automate.pm

    You could also set things up so that you have full-blown inheritance, so that your end-user directly asks for a sub object, and the lower-level items pull in methods and configuration based on its parents, and things may look differently.

    When designing something like this that is *bound* to expand exponentially, laying it out in a way that will be extensible is paramount. My suggestion is to start small, use Module::Starter to get set up, muck about, then start your unit tests. While writing tests, you'll quickly find out if your layout and/or your structure makes sense for going forward.

    There is no one answer for this, as you are very quickly going to realize. Test, play, scrap, test, play scrap.

    The one piece of advice I do have here that my fellow Monks will all agree on, is use a Version Control System (VCS). Github is free, as is Bitbucket. Spending the time to learn git (or hg or svn etc) now will save you enormous amounts of time going forward.

      Thank you very much. I didn't have much trouble finding the documentation to write my first module, but finding guidelines on how to organize modules in relation to each other has proven more elusive. <cue the flood gates!> And now that I've rolled one by hand, I'll definitely give Module::Starter a try. So, thank you again.

      Version Control. Yes. I'm a zealot actually. I've been religiously stuffing my whole life into various VCS's since the days when RCS was an exciting improvement to SCCS. :)
      But always good advice, and never over stated.

      I do have another question about the integration of my module hierarchy into the greater CPAN tree, but I'll follow up after I've done a little more research and can ask an intelligent question.

      Thanks again,
      cbeckley

Re: RFC and Questions regarding a module to automate and centralize infrastructure administration
by stevieb (Canon) on Mar 25, 2017 at 02:39 UTC
    " For instance, the line where I access the array of commands: for my $cmd (@{$ctxt->{commands}}) { ..."

    That depends. In one of my distributions where I knew extensibility would be required, I built an "engine" type system. It was filter based, so I had these phases:

    - pre processor - processor - post processor - engine

    That compromised the "core" of the project. How I designed it was that the engine was a part of the core, where the "processor" phase was at its root (the absolute core), and input to it needed to be in an expected format, and output from it would be in an expected format. What the pre-proc did before it reached the proc phase was irrelevant (so long as the proc phase got the proper formatted data), and the output from the post proc and engine was irrelevant as well, as the upper-level methods would output the data, according to the documentations spec.

    How I accomplished this, was to create hash tables containing subroutine references to various functionality methods, then 'register' those routines within the main object.

    Here's a public method I made available for development and convenience that lists the non-private 'registered' engine methods:

    sub engines { trace() if $ENV{TRACE}; my $self = shift; my $module = $self->{namespace} . "::Engine"; my $engine = $module->new; my @engines; for (keys %{$engine->_dt}){ push @engines, $_ if $_ !~ /^_/; } return @engines; }

    The Engine module initialization code (new()), along with the dispatch table that links a command to an actual function:

    sub new { trace() if $ENV{TRACE}; my $self = {}; bless $self, shift; $self->{engines} = $self->_dt; return $self; } sub _dt { trace() if $ENV{TRACE}; my $self = shift; my $dt = { all => \&all, has => \&has, missing => \&missing, lines => \&lines, objects => \&objects, search_replace => \&search_replace, inject_after => \&inject_after, dt_test => \&dt_test, _test => \&_test, _test_bad => \&_test_bad, }; return $dt; }

    ...here's an example of an Engine itself:

    sub all { trace() if $ENV{TRACE}; return sub { trace() if $ENV{TRACE}; my $p = shift; my $struct = shift; my $file = $p->{file}; my @subs; for my $name (@{ $p->{order} }){ push @subs, grep {$name eq $_} keys %{ $struct->{$file}{su +bs} }; } return \@subs; }; }

    ...and finally, the _core() code that looks for the engine specific code and loads it, whether it's an already registered engine, or a user-supplied code reference of a new/custom one:

    sub _engine { trace() if $ENV{TRACE}; my $self = shift; my $p = shift; my $struct = shift; my $engine = defined $p->{engine} ? $p->{engine} : $self->{params}{engine}; if (not $engine or $engine eq ''){ return $struct; } my $cref; if (not ref($engine) eq 'CODE'){ # engine is a name my $engine_module = $self->{namespace} . "::Engine"; my $compiler = $engine_module->new; # engine isn't in the dispatch table if (! $compiler->exists($engine)){ confess "engine '$engine' is not implemented.\n"; } eval { $cref = $compiler->{engines}{$engine}->(); }; # engine has bad func val in dispatch table, but key is ok if ($@){ $@ = "\n[Devel::Examine::Subs speaking] " . "dispatch table in Devel::Examine::Subs::Engine " . "has a mistyped function as a value, but the key is +ok\n\n" . $@; confess $@; } } if (ref($engine) eq 'CODE'){ $cref = $engine; } if ($self->{params}{engine_dump}){ my $subs = $cref->($p, $struct); print Dumper $subs; exit; } return $cref; }

    Now, when someone calls the all() method on the main object, this is what is executed:

    sub all { trace() if $ENV{TRACE}; my $self = shift; my $p = $self->_params(@_); $self->{params}{engine} = 'all'; $self->run($p); }

    ...run() is the method that actually instigates the whole process. The above all() is the most simplistic as it gets. If more phases were involved, this method kicks it off, it'll process through all of the phases, then gets it back for return to the method that originally called it for return to the user:

    sub run { trace() if $ENV{TRACE}; my $self = shift; my $p = shift; $self->_config($p); $self->_run_end(0); my $struct; if ($self->{params}{directory}){ $struct = $self->_run_directory; } else { $struct = $self->_core; $self->_write_file if $self->{write_file_contents}; } $self->_run_end(1); return $struct; }

    Then, when _core() is called, it iterates through the phases, calling each one at a time (if they are needed), and in this case, it'll eventually see an engine is located. This is how that phase is executed. $engine is the subroutine callback (code reference) specified, and that's how we can call it as a function like below.

    my $engine = $self->_engine($p, $subs); if ($self->{params}{engine}){ $subs = $engine->($p, $subs); $self->{write_file_contents} = $p->{write_file_contents}; }

    After that, there's further processing, but since "engine" is the last phase, eventually, $subs gets put back into $struct, and when things are done, that is what is returned back to run(), which is returned back to all() which is returned back to the user caller.

    So, that was kind of fun going back through some of the code I wrote that I had fun designing that allowed the shifting in and out of code blocks for different purposes.

    That's just one way to make things a bit modular. Note, I said one way. There are many, and the deeper and more complex you get, the more convoluted your code may become, unless you have a great test suite, and a whole crapload of good documentation. Those two things are key to building a system that you're looking at approaching :)

    fwiw, the code above belongs to my Devel::Examine::Subs.

    update: Good thing is when you go back and examine your code a few years later, you can see a whole lot of "NOPE" that you wouldn't do now and it looks out of place, which means refactoring ;)

Re: RFC and Questions regarding a module to automate and centralize infrastructure administration
by melezhik (Monk) on Mar 25, 2017 at 07:49 UTC
    "Had this been done before." -

    Hi! Probably Rex might be of interest to you. It is is pure Perl remote execution engine targeted for bulk servers operations.

        Definitely, definitely, definitely avoid Net::FullAuto.

        More context.

        Thank you both, I will certainly give Rex a look.

        I can't overstate the value this forum provides regarding module advice.

        First, for the suggestions of applicable modules. This isn't the first time I've come up dry searching CPAN both through CPAN and google.

        Beyond that, and far more valuable, the inside knowledge that comes along for the ride: "theres X and Y but avoid Y" or "X, Y, and Z, but Z is unsupported and only Y installs easily on your OS."

        Amazing.

        Thanks,
        cbeckley

Re: RFC and Questions regarding a module to automate and centralize infrastructure administration
by salva (Abbot) on Mar 25, 2017 at 07:42 UTC
    I've written a module, nascent, that makes it a little easier to execute a list of commands on a group of servers. Basically I create an array of servers, which I refer to as contexts in the code, each of which has an array of commands and all the commands get executed in all the contexts and all the output is gathered and returned for processing.

    That's very similar to what Net::OpenSSH::Parallel does. The mayor difference is that it doesn't capture the output directly, you have to save it into files that you can read afterwards for processing.

      Thank you very much for the suggestion. I've looked through Net::OpenSSH::Parallel now, and it does seem to do a lot of what I'm looking for. I've always been rather averse to leaving execution artifacts behind, but if it gets the job done, I'll get over it!

      Thanks again for taking the time.

      Thanks,
      cbeckley

        Initially, I did not include in the module the capacity to capture output directly in order to simplify its implementation and also because I was not able to imagine an API supporting it that would make sense while still being simple to learn an use...

        Over time, I have come to appreciate that way of working. Using the file-system to store the generated data escalates quite well, allows easier debugging and in most cases, the performance penalty is negligible (usually the bottleneck is in the network). Also, the mechanism is easy to understand and to use.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1185850]
Approved by johngg
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (7)
As of 2020-06-01 16:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?



    Results (4 votes). Check out past polls.

    Notices?