http://www.perlmonks.org?node_id=865339

pileofrogs has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, Monks through whom perl doth rock.

I'm a sysadmin and I recently got bitten by a script that sent me an SMS message every minute for several days leading to a $300 phone bill. Before now, it's only been filling up the inbox, but that sucks too.

What is the best way to limit alert messages? I'm sure there are many reasonable approaches and some of you have probably already given this problem some thought.

Thanks!
--Pileofrogs

Replies are listed 'Best First'.
Re: Module to limit email floods?
by lostjimmy (Chaplain) on Oct 14, 2010 at 19:37 UTC
    A simple approach would be to have your script create a digest of alerts and only send an actual email or SMS at a specified interval.

    To further condense the information, if at each interval you only care that a specific alert has occurred, not how many times or at what time, you could just store the unique alerts that were received and send those.

Re: Module to limit email floods?
by Corion (Patriarch) on Oct 14, 2010 at 19:30 UTC

    I'd look at Algorithm::TokenBucket for the fancy variant of rate limiting. Alternatively, if it's just simple lines, maybe you can set up syslogd to send you an email/SMS for a certain category? I think syslogd has its own limiting, at least for quickly repeating messages.

Re: Module to limit email floods?
by hbm (Hermit) on Oct 14, 2010 at 20:55 UTC

    I assume it is your script, and it is cron'd? I used to monitor roughly like this:

    if (problem) { fixIt; logIt; if (-f $indicator) { pageMe; unlink $indicator; } else { touch $indicator; } } else { unlink $indicator; }

    Thus, I'd get paged only if a problem persisted. With reliable "fixIt" and detection methods, and an appropriate cron schedule, that served me well.

    If your condition isn't programmatically fixable, I'd change that around to perhaps this:

    if (problem) { logIt; if (! -f $indicator) { pageMe; touch $indicator; } } else { unlink $indicator; }

    Thus, you'd only get paged initially. Or you might also page yourself when the error is cleared; or check the timestamp of the indicator and page every hour; or whatever.

    (I remember a similar bill when first testing such scripts, despite "unlimited" texts on my pager...)

     

    Update: The best way to limit alerts is to limit alert conditions...
    pileofrogs, what was the condition that lasted several days? How was it resolved? Could it not have been resolved automatically (with perl)?

      My problem with all these different methods of rate-limiting is that there is no One True Way, even within a single system.

      You may want to only get a single message for one kind of error condition, but for some others you may want periodic messages, and for some you might even want to have the messages get escalated to some other person or use some other method of notification.

      What we really need is a completely separate, simple, configurable notification system, with an API, web interface and command-line tools. If done properly, any other system that needed notification-sending could use it.

      Unfortunately, to be really effective, it needs to know the current status of "things", which is getting into the realm of monitoring. I don't see the various monitoring systems changing to use this hypothetical notification system, so the only way I can see to make this work would be to include adapters for various monitoring systems which would continuously import state. Then the notifications could be triggered off it. That would be a lot to do, hoping that it would be used.

        I think you described log4perl (or whatever) + syslog + nagios , maybe
Re: Module to limit email floods?
by BrowserUk (Patriarch) on Oct 14, 2010 at 21:46 UTC

    Wrap the sub that sends the alerts in a sub that records and checks for the last time that particular alert was sent, and have it do nothing if it was already sent within some specified period:

    use constant INTERVAL => 3600; my %alerts; sub wrapSendSMS{ my( $alert, $no ) = @_; ## Do nothing if we've sent one recently return if exists $alerts{ $alert } and time() - $alerts{ $alert } < INTERVAL; ## Send it and record when sendSMS( $alert, $no ); $alerts{ $alert } = time; return 1; }

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Module to limit email floods?
by dwm042 (Priest) on Oct 14, 2010 at 21:28 UTC
    The best way to limit alert messages and/or email alerts is to not use them, as they don't scale.

    Monitoring tools with web interfaces do scale. They also allow you to drill down, to find out just when a problem happened, when you can actually respond to the problem.

      But "getting an email" is passive while "web interfaces" is active. And believe me, people prefer to be passive over being active, at least when it comes to looking after things.

      Thats a nice idea that will not work, as I assume the reason for sending the SMS alert is to be able to leave the house and not have to look at a screen all the time.

      I would still suggest looking into a Nagios like solution that deals with all the monitoring, shows the cool webpage and has the ability to contact you via SMS, email or both. It is a very mature tool that has long since dealt with this and many other problems you will face when monitoring a system or systems. Pretty much every single major soft and hardware solution has modules available to monitor the solutions specifics which will make your life so much easier. Writing your own monitoring solution is really simple and should allow you to save a lot of time as all you need to do is let the central system know about your findings all alerting is dealth with by the central system so no worries there

        RatKing,

        It's far more than a nice idea that will not work. Just work in a place where hundreds of scripts send out thousands of emails a day -- I've had more than one job with those parameters -- and the alternative is not only desirable but eventually necessary. Doing systems administration via email trail is a way to waste alerts, fill email boxes and waste admin time.

        No on-the-job script should ever send out an email alert. It should alert a monitoring tool, such as the aforementioned Nagios. A higher end layer of software, or a help desk, can decide to alert admins *if necessary*. This is the kind of thing that a smart shop can automate and a foolish shop will drown admins in electronic paper.

        Things get missed in floods of spam. So the smart thing to do is stop the spamming, consolidate the reporting, and push the alerting to a decision making layer. Status messages can then be consolidated in a web interface, as they should be.

Re: Module to limit email floods?
by davies (Prior) on Oct 14, 2010 at 21:43 UTC
    SpamGourmet creates almost infinite personalised addresses which get forwarded to a "hidden" address. These are limited unless the user takes some action. The source code (Perl, naturally) is available from SourceForge.

    Regards,

    John Davies
Re: Module to limit email floods?
by aquarium (Curate) on Oct 15, 2010 at 05:38 UTC
    there are plenty of monitoring systems around, free and/or commercial, where you can have SMS/email/Web alerts and drill down etc. they're usually smart enough with regard to messaging, and can be setup to strike a balance between alerting you of a major problem and not becoming pesky.
    i try to stay away from hand rolled solutions for both monitoring and automatically fixing stuff (yikes!)
    otherwise you can get blamed for not only not attending to the problem (which you still have to fix), but now you also need to resolve the problem with the monitor/fix script. in other words, over-automation can cause more problems than it tries to fix.
    anyway, that's my stance on it and you may not agree.
    have fun sysadmining.
    the hardest line to type correctly is: stty erase ^H
Re: Module to limit email floods?
by sundialsvc4 (Abbot) on Oct 15, 2010 at 17:13 UTC

    I have not personally looked into this, but surely there must be “apps” for various smart-phones that, in the various appropriate ways for each type of phone, implement “push notifications” and provide you the means to flip through them.   For instance, the ubiquitous “Facebook” app.   You get a notification that there is something out there that you need to look at, and then, with an appropriate app, you can retrieve and browse-through the list.

    Neither e-mail nor SMS are (today) the best way to accomplish this task.