Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: RFC: A module for SNMP polling of thousands of nodes simultaneously

by Hercynium (Hermit)
on Jan 22, 2008 at 05:38 UTC ( #663506=note: print w/ replies, xml ) Need Help??


in reply to Re: RFC: A module for SNMP polling of thousands of nodes simultaneously
in thread RFC: A module for SNMP polling of thousands of nodes simultaneously

My current usage of SNMP::Effective may not be as optimal as it could. I'm going to try again with some better-tuned code, but here's what I've got so far...

(please note - I haven't verified if the SNMP query results are correct - it's rather late and I'm bushed, so I'm making a big assumption that they are...)

#!/usr/bin/perl use strict; use warnings; use Data::Dumper; use Carp; use Parse::CSV; use SNMP; use SNMP::Effective; #--------------------------------------------------------- my $csv_file = shift || die "Please specify a CSV file with SNMP host +info!"; my @reqired_fields = qw(HOSTIP COMMUNITY SNMPVER SNMPPORT); my @hosts = read_hosts_csv($csv_file, @reqired_fields); # This object encapsulates the desired queries to run. my $query = SNMP::Effective->new( max_sessions => 50, #master_timeout => $TIMEOUT_SECONDS, ); my @results; foreach my $host (@hosts) { my @varbinds = qw( ifDescr ifInOctets ifOutOctets ifAlias ifType ifName ifInErrors ifOutErrors ifSpeed ifAdminStatus ifOperStatus ); my $callback = sub { my $host_obj = shift; # SNMP::Effective::Host object my $error = shift; # Error message. if ($error) { warn $error; return; } push @results, [ $host_obj->address(), $host_obj->data() ]; }; $query->add( dest_host => $host->{HOSTIP}, callback => $callback, walk => [ @varbinds ], arg => { Community => $host->{COMMUNITY}, Version => $host->{SNMPVER}, RemotePort => $host->{SNMPPORT}, }, ); } # Run all the added queries with up to X # asynchronous operations in-flight at any time. foreach my $iter (1..1) { sleep 30 unless $iter == 1; $query->execute(); print Dumper \@results; } exit; #--------------------------------------------------------- # Read in the CSV file. sub read_hosts_csv { my $file = shift; my @required_fields = @_; # Parse entries from a CSV file into hashes hash my $csv_parser = Parse::CSV->new( file => $file, fields => 'auto', # Use the first line as column headers, # which become the hash keys. ); my @node_cfg; # Return a reference to this my $line_num = 0; while ( my $line = $csv_parser->fetch() ) { $line_num++; my $error_flag = 0; foreach my $field (@required_fields) { if ( ! exists $line->{$field} ) { $error_flag = 1; carp "Missing field [$field] on line [$line_num] in CS +V file [$file]"; } } croak "Terminating due to errors on line [$line_num] in CSV fi +le [$file]" if $error_flag; push @node_cfg, $line; } if ( $csv_parser->errstr() ) { croak "Fatal error parsing [$file]: " . $csv_parser->errstr(); } return @node_cfg; } 1; __END__

The average run-time for the same 1000 nodes is 125 - 130 seconds, at about 25% CPU, using 50 simultaneous sessions.

It's certainly not an order of magnitude of difference, however, on a smaller set of nodes, like 100, both SNMP::Effective and my own code are neck-in-neck at about 7 seconds. This leads me to hypothesize that my code *may* scale better as-it-is, but that theory will require more testing and information. Also, as I (or other hackers) find bugs, add features, etc... the scaling factors may change enough that SNMP::Session will truly be the faster of the two.

Right now, I still maintain that SNMP::Effective is substantially more mature, robust, and well-tested than what I've written... but it currently seems that *I* am faster :)


Comment on Re^2: RFC: A module for SNMP polling of thousands of nodes simultaneously
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://663506]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (6)
As of 2015-07-05 22:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (68 votes), past polls