Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re^2: RFC: A module for SNMP polling of thousands of nodes simultaneously

by Hercynium (Hermit)
on Jan 22, 2008 at 05:38 UTC ( #663506=note: print w/replies, xml ) Need Help??

in reply to Re: RFC: A module for SNMP polling of thousands of nodes simultaneously
in thread RFC: A module for SNMP polling of thousands of nodes simultaneously

My current usage of SNMP::Effective may not be as optimal as it could. I'm going to try again with some better-tuned code, but here's what I've got so far...

(please note - I haven't verified if the SNMP query results are correct - it's rather late and I'm bushed, so I'm making a big assumption that they are...)
#!/usr/bin/perl use strict; use warnings; use Data::Dumper; use Carp; use Parse::CSV; use SNMP; use SNMP::Effective; #--------------------------------------------------------- my $csv_file = shift || die "Please specify a CSV file with SNMP host +info!"; my @reqired_fields = qw(HOSTIP COMMUNITY SNMPVER SNMPPORT); my @hosts = read_hosts_csv($csv_file, @reqired_fields); # This object encapsulates the desired queries to run. my $query = SNMP::Effective->new( max_sessions => 50, #master_timeout => $TIMEOUT_SECONDS, ); my @results; foreach my $host (@hosts) { my @varbinds = qw( ifDescr ifInOctets ifOutOctets ifAlias ifType ifName ifInErrors ifOutErrors ifSpeed ifAdminStatus ifOperStatus ); my $callback = sub { my $host_obj = shift; # SNMP::Effective::Host object my $error = shift; # Error message. if ($error) { warn $error; return; } push @results, [ $host_obj->address(), $host_obj->data() ]; }; $query->add( dest_host => $host->{HOSTIP}, callback => $callback, walk => [ @varbinds ], arg => { Community => $host->{COMMUNITY}, Version => $host->{SNMPVER}, RemotePort => $host->{SNMPPORT}, }, ); } # Run all the added queries with up to X # asynchronous operations in-flight at any time. foreach my $iter (1..1) { sleep 30 unless $iter == 1; $query->execute(); print Dumper \@results; } exit; #--------------------------------------------------------- # Read in the CSV file. sub read_hosts_csv { my $file = shift; my @required_fields = @_; # Parse entries from a CSV file into hashes hash my $csv_parser = Parse::CSV->new( file => $file, fields => 'auto', # Use the first line as column headers, # which become the hash keys. ); my @node_cfg; # Return a reference to this my $line_num = 0; while ( my $line = $csv_parser->fetch() ) { $line_num++; my $error_flag = 0; foreach my $field (@required_fields) { if ( ! exists $line->{$field} ) { $error_flag = 1; carp "Missing field [$field] on line [$line_num] in CS +V file [$file]"; } } croak "Terminating due to errors on line [$line_num] in CSV fi +le [$file]" if $error_flag; push @node_cfg, $line; } if ( $csv_parser->errstr() ) { croak "Fatal error parsing [$file]: " . $csv_parser->errstr(); } return @node_cfg; } 1; __END__

The average run-time for the same 1000 nodes is 125 - 130 seconds, at about 25% CPU, using 50 simultaneous sessions.

It's certainly not an order of magnitude of difference, however, on a smaller set of nodes, like 100, both SNMP::Effective and my own code are neck-in-neck at about 7 seconds. This leads me to hypothesize that my code *may* scale better as-it-is, but that theory will require more testing and information. Also, as I (or other hackers) find bugs, add features, etc... the scaling factors may change enough that SNMP::Session will truly be the faster of the two.

Right now, I still maintain that SNMP::Effective is substantially more mature, robust, and well-tested than what I've written... but it currently seems that *I* am faster :)
  • Comment on Re^2: RFC: A module for SNMP polling of thousands of nodes simultaneously
  • Download Code

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://663506]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (3)
As of 2017-01-16 20:24 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (151 votes). Check out past polls.