RFC: A module for SNMP polling of thousands of nodes simultaneouslyby Hercynium (Hermit)
|on Jan 15, 2008 at 18:42 UTC||Need Help??|
After some trial and error, as well as trying every alternative I could find, I've written a new SNMP module that allows for fast asynchronous polling of multiple objects across thousands of nodes.
My tests across 1000 nodes at my employer's network have so far been *very* successful! The usage example completes one round of polling of these nodes in under 30 seconds, with about 25% CPU utilization. If I increase the InFlight parameter to 50, it completes in under 20 seconds with 37% CPU!
At this time, the module is the bare-minimum necessary for my needs, but I believe it has the potential for much, much more. I won't go into the details here of why I've written my own but suffice to say that nothing had the right combination of features, stability, and speed that I needed.
The code makes heavy use of closures, and I'm fairly new to the concept. It depends on the net-snmp libraries, and the SNMP module that comes with them.
IMPORTANT -- in order for this to work, you must be using the SNMP module from net-snmp release version 5.3.2 or the latest version from CVS trunk. The release version 5.4.1 (as well as the versions from CPAN) has a bug in the XS code that leads to a core-dump when running too many asynchronous queries in parallel.
What I want now
I'm posting this to the monastery in hopes that the knowledge of the Monks can help me make further improvemens, and find and fix bugs and any sub-optimal design decisions. This includes:
UPDATE: I've posted a new version of the code here. The code in this node should now be considered obsolete. The new code is slightly faster, more robust, and *much* more scalable. It also has new features.
The old code is still here, but is now hidden with spoiler tags. I welcome any and all comments and criticism!