Re^2: RFC: A module for SNMP polling of thousands of nodes simultaneously

[Hercynium]

My current usage of SNMP::Effective may not be as optimal as it could. I'm going to try again with some better-tuned code, but here's what I've got so far...

(please note - I haven't verified if the SNMP query results are correct - it's rather late and I'm bushed, so I'm making a big assumption that they are...)

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;
use Carp;
use Parse::CSV;
use SNMP;
use SNMP::Effective;


#---------------------------------------------------------

my $csv_file = shift || die "Please specify a CSV file with SNMP host 
+info!";

my @reqired_fields = qw(HOSTIP COMMUNITY SNMPVER SNMPPORT);
my @hosts = read_hosts_csv($csv_file, @reqired_fields);


# This object encapsulates the desired queries to run.
my $query = SNMP::Effective->new(
    max_sessions   => 50,
    #master_timeout => $TIMEOUT_SECONDS,
);

my @results;
foreach my $host (@hosts) {

    my @varbinds = qw(
           ifDescr ifInOctets ifOutOctets ifAlias ifType
           ifName ifInErrors ifOutErrors ifSpeed
           ifAdminStatus ifOperStatus
    );

    my $callback = sub {
        my $host_obj  = shift; # SNMP::Effective::Host object
        my $error     = shift; # Error message.
        if ($error) {
            warn $error;
            return;
        }
        push @results, [ $host_obj->address(), $host_obj->data() ];
    };

    $query->add(
        dest_host => $host->{HOSTIP},
        callback  => $callback,
        walk      => [ @varbinds ],
        arg       => {
            Community  => $host->{COMMUNITY},
            Version    => $host->{SNMPVER},
            RemotePort => $host->{SNMPPORT},
        },
    );

}

# Run all the added queries with up to X
# asynchronous operations in-flight at any time.
foreach my $iter (1..1) {
    sleep 30 unless $iter == 1;
    $query->execute();
    print Dumper \@results;
}


exit;
#---------------------------------------------------------

# Read in the CSV file.
sub read_hosts_csv {
    my $file = shift;
    my @required_fields = @_;

    # Parse entries from a CSV file into hashes hash
    my $csv_parser = Parse::CSV->new(
        file   => $file,
        fields => 'auto',  # Use the first line as column headers,
                           # which become the hash keys.
    );

    my @node_cfg; # Return a reference to this
    my $line_num = 0;
    while ( my $line = $csv_parser->fetch() ) {
        $line_num++;
        my $error_flag = 0;
        foreach my $field (@required_fields) {
            if ( ! exists $line->{$field} ) {
                $error_flag = 1;
                carp "Missing field [$field] on line [$line_num] in CS
+V file [$file]";
            }
        }
        croak "Terminating due to errors on line [$line_num] in CSV fi
+le [$file]"
            if $error_flag;

        push @node_cfg, $line;
    }

    if ( $csv_parser->errstr() ) {
        croak "Fatal error parsing [$file]: " . $csv_parser->errstr();
    }

    return @node_cfg;
}


1;
__END__

The average run-time for the same 1000 nodes is 125 - 130 seconds, at about 25% CPU, using 50 simultaneous sessions.

It's certainly not an order of magnitude of difference, however, on a smaller set of nodes, like 100, both SNMP::Effective and my own code are neck-in-neck at about 7 seconds. This leads me to hypothesize that my code *may* scale better as-it-is, but that theory will require more testing and information. Also, as I (or other hackers) find bugs, add features, etc... the scaling factors may change enough that SNMP::Session will truly be the faster of the two.

Right now, I still maintain that SNMP::Effective is substantially more mature, robust, and well-tested than what I've written... but it currently seems that *I* am faster :)