Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Simple Perl NMS Polling Engine

by mmoorreett (Acolyte)
on Jun 27, 2020 at 02:41 UTC ( #11118591=perlquestion: print w/replies, xml ) Need Help??

mmoorreett has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks

I'm looking for some high level/framework advice on how to best develop a solution to build a *simple* network polling engine. I'm not looking to re-invent the wheel or want to implement a full blown NMS, just build something thats light weight and fast/efficient to plug a gap we have at the moment. Since being tasked with this, I have researched various frameworks I have not used before for multitasking/parallelism. E.g, threads, forks, event loop frameworks, such as POE, AnyEvent, IO:Async. I've used Perl periodically for various applications but not for something like this. I've started playing with the mentioned modules/frameworks and developed some samples, but I am unsure what is the best one. At a high level I need to:

Periodically query an external data source for device data

ICMP Ping poll on a configurable schedule (5 minute), potentially a couple of thousand devices, in parallel, return results for further processing

Separate to ICMP poll, collect SNMP data on a configurable schedule (5 minute), potentially a couple of thousand devices, in parallel, return results for further processing

Each poll thread/cycle needs a "TTL", that is, each poll thread must complete by X seconds, (not talking about an ICMP/SNMP timeout)

Needs to run on Windows and Nix perl.

Not sure what the best approach to take is. Having looked at a couple of open source NMS solutions i found built in perl, threads/forks were used, but these are over a decade old so I'm not sure if the event loop frameworks were considered or suitable.

Looking to get some guidance on the most appropriate path to take so I don't need to re-invent the wheel, again :).

Thanks in advance.

Replies are listed 'Best First'.
Re: Simple Perl NMS Polling Engine
by parv (Vicar) on Jun 27, 2020 at 08:39 UTC
      although its part of the same piece of work, it is different, here i'm asking about the holistic approach i should take
Re: Simple Perl NMS Polling Engine
by hippo (Chancellor) on Jun 27, 2020 at 07:33 UTC
    Each poll thread/cycle needs a "TTL", that is, each poll thread must complete by X seconds, (not talking about an ICMP/SNMP timeout)

    What happens if it doesn't complete by X seconds?

Re: Simple Perl NMS Polling Engine
by perlfan (Priest) on Jun 27, 2020 at 06:52 UTC
    In terms of scaling, using a consumer/producer worker queue model is always a consideration. I like redis, personally and it can be used either with your own worker daemons distributed however or via one of the async frameworks you mentioned.

    Your mentioning of SNMP made me look to see if observium had some sort of API and Perl client, but instead I found App::Netdisco - the author may have solved some or all of the problem you're addressing. I found it because it does have an observium plugin, App::NetdiscoX::Web::Plugin::Observium. Looking at the author profile on metacpan, I also saw: YATG (Fast SNMP data poller daemon, with storage and graphing) and YATG::Tutorial.

      Thanks, yes I did see App::Netdisco but not YATG, YATG looks to be more inline with what i need so i'll check that out, thanks

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11118591]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2020-07-06 07:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?