Re: RFC and Questions regarding a module to automate and centralize infrastructure administration

[stevieb]

" For instance, the line where I access the array of commands: for my $cmd (@{$ctxt->{commands}}) { ..."

That depends. In one of my distributions where I knew extensibility would be required, I built an "engine" type system. It was filter based, so I had these phases:

- pre processor
- processor
- post processor
- engine

That compromised the "core" of the project. How I designed it was that the engine was a part of the core, where the "processor" phase was at its root (the absolute core), and input to it needed to be in an expected format, and output from it would be in an expected format. What the pre-proc did before it reached the proc phase was irrelevant (so long as the proc phase got the proper formatted data), and the output from the post proc and engine was irrelevant as well, as the upper-level methods would output the data, according to the documentations spec.

How I accomplished this, was to create hash tables containing subroutine references to various functionality methods, then 'register' those routines within the main object.

Here's a public method I made available for development and convenience that lists the non-private 'registered' engine methods:

sub engines {
     
    trace() if $ENV{TRACE};
 
    my $self = shift;
    my $module = $self->{namespace} . "::Engine";
    my $engine = $module->new;
  
    my @engines;
 
    for (keys %{$engine->_dt}){
        push @engines, $_ if $_ !~ /^_/;
    }
    return @engines;
}

The Engine module initialization code (new()), along with the dispatch table that links a command to an actual function:

sub new {
     
    trace() if $ENV{TRACE};
 
    my $self = {};
    bless $self, shift;
 
    $self->{engines} = $self->_dt;
 
    return $self;
}
sub _dt {
     
    trace() if $ENV{TRACE};
 
    my $self = shift;
 
    my $dt = {
        all => \&all,
        has => \&has,
        missing => \&missing,
        lines => \&lines,
        objects => \&objects,
        search_replace => \&search_replace,
        inject_after => \&inject_after,
        dt_test => \&dt_test,
        _test => \&_test,
        _test_bad => \&_test_bad,
    };
 
    return $dt;
}

...here's an example of an Engine itself:

sub all {
     
    trace() if $ENV{TRACE};
 
    return sub {
         
        trace() if $ENV{TRACE};
 
        my $p = shift;
        my $struct = shift;
 
        my $file = $p->{file};
 
        my @subs;
 
        for my $name (@{ $p->{order} }){
            push @subs, grep {$name eq $_} keys %{ $struct->{$file}{su
+bs} };
        } 
 
        return \@subs;
    };
}

...and finally, the _core() code that looks for the engine specific code and loads it, whether it's an already registered engine, or a user-supplied code reference of a new/custom one:

sub _engine {
     
    trace() if $ENV{TRACE};
 
    my $self = shift;
    my $p = shift;
    my $struct = shift;
 
    my $engine
      = defined $p->{engine} ? $p->{engine} : $self->{params}{engine};
 
    if (not $engine or $engine eq ''){
        return $struct;
    }
 
    my $cref;
 
    if (not ref($engine) eq 'CODE'){
 
        # engine is a name
 
        my $engine_module = $self->{namespace} . "::Engine";
        my $compiler = $engine_module->new;
 
        # engine isn't in the dispatch table
 
        if (! $compiler->exists($engine)){
            confess "engine '$engine' is not implemented.\n";
        }
 
        eval {
            $cref = $compiler->{engines}{$engine}->();
        };
 
        # engine has bad func val in dispatch table, but key is ok
 
        if ($@){
            $@ = "\n[Devel::Examine::Subs speaking] " .
                  "dispatch table in Devel::Examine::Subs::Engine " .
                  "has a mistyped function as a value, but the key is 
+ok\n\n"
            . $@;
            confess $@;
        }
    }
 
    if (ref($engine) eq 'CODE'){
        $cref = $engine;
    }
 
    if ($self->{params}{engine_dump}){
        my $subs = $cref->($p, $struct);
        print Dumper $subs;
        exit;
    }
 
    return $cref;
}

Now, when someone calls the all() method on the main object, this is what is executed:

sub all {
     
    trace() if $ENV{TRACE};
 
    my $self = shift;
    my $p = $self->_params(@_);
 
    $self->{params}{engine} = 'all';
     
    $self->run($p);
}

...run() is the method that actually instigates the whole process. The above all() is the most simplistic as it gets. If more phases were involved, this method kicks it off, it'll process through all of the phases, then gets it back for return to the method that originally called it for return to the user:

sub run {
 
    trace() if $ENV{TRACE};
 
    my $self = shift;
    my $p = shift;
 
    $self->_config($p);
 
    $self->_run_end(0);
 
    my $struct;
 
    if ($self->{params}{directory}){
        $struct = $self->_run_directory;
    }
    else {
        $struct = $self->_core;
        $self->_write_file if $self->{write_file_contents};
    }
 
    $self->_run_end(1);
 
    return $struct;
}

Then, when _core() is called, it iterates through the phases, calling each one at a time (if they are needed), and in this case, it'll eventually see an engine is located. This is how that phase is executed. $engine is the subroutine callback (code reference) specified, and that's how we can call it as a function like below.

    my $engine = $self->_engine($p, $subs);
 
    if ($self->{params}{engine}){
        $subs = $engine->($p, $subs);
        $self->{write_file_contents} = $p->{write_file_contents};
    }

After that, there's further processing, but since "engine" is the last phase, eventually, $subs gets put back into $struct, and when things are done, that is what is returned back to run(), which is returned back to all() which is returned back to the user caller.

So, that was kind of fun going back through some of the code I wrote that I had fun designing that allowed the shifting in and out of code blocks for different purposes.

That's just one way to make things a bit modular. Note, I said one way. There are many, and the deeper and more complex you get, the more convoluted your code may become, unless you have a great test suite, and a whole crapload of good documentation. Those two things are key to building a system that you're looking at approaching :)

fwiw, the code above belongs to my Devel::Examine::Subs.

update: Good thing is when you go back and examine your code a few years later, you can see a whole lot of "NOPE" that you wouldn't do now and it looks out of place, which means refactoring ;)