RFC: Module idea: Carp + controlled email

[pileofrogs]

I'm a network admin, and I use perl to automate just about everything on my systems. A lot of the time, I want my scripts to email me if conditions are bad and page me if conditions are terrible. I've experiemented with a few different mechanisms and I think I'm almost ready to write a real module.

I've looked at Carp::Notify by Jim Thomason and my idea is significantly different so I don't think I'd just be rebuilding his wheel.

Here's what I want my module to do:

1)read contact info from a central file: (/etc/contacts or something like that.) This eliminates the need to update all my scripts when I or my boss gets a new pager number.

2)Control how often emails get sent: I've sent myself 1000 emails in a minute before. Don't want that happening.

3)Optionally override normal warn/die/carp/croak calls: I'd like to be able to add this module to existing scripts and have it start emailing me about the events that I DON'T anticipate.

4)Optionally handle unexpected death as well as die/croak calls: I want to be notified in every case, not just the ones I thought of. If my CGI devides by zero or doesn't compile, I want to know about it.

5)Make it so the user can do as much configuration as possible at the beginning of his/her script, leaving actual warn/carp/croak/die/whateverIcallthem calls as simple as possible. I want to make these calls a LOT so I want them to be simple.

I've already done all these bits in different places. All I need to do is get it togeather in one module. Making it portable will probably be most of the challange.

The only thing I do that isn't either straightforward or just cribbed from someone elses's module is the email limiting, so I'll mention how I do that here.

Every time I send an email, I make an md5 hash of the recipients + the subject. I then make an entry in a GDBM file with that hash pointing toward a timestamp. The timestamp is the time when I consider it OK to send the same email again. So, before the email gets sent, it checks the md5 hash above against the GDBM file to see if it's been sent before, and if it has if the ok-to-send-now-timestamp is either right now or in the past.

So, oh monks of perl, do you have any suggestions for such a module? Do you think it's a good idea? Do you think it's a bad idea, and I should go soak my head?

I'm thinking of calling it Carp::Fancy. Any suggestions on better ideas for a name would also be great.

Thanks
-Dylan (aka Pileofrogs)

[tilly]

You seem to be mixing 2 pieces of functionality that I would seperate.

1. You want problems to get recorded somewhere.

2. You want to hear about problems that are happening.

I would solve your problem by making sure that everything was using some sort of centralized logging. I would then have a cron job that processes the logs and decides when to send you email based on it.

That way when something is borked, you won't have a thousand processes a minute trying to decide whether or not to send an email. You'll also have the opportunity to be able to summarize things in a more flexible way. For instance if you see that some obscure error happened, is it a priority or not? Maybe, maybe not. If you you get a message that it happened 1000 times in the last minute, is it a priority or not? You betcha!

Another, more subtle, benefit. What happens if you have your special logic and there is some system issue that causes an error when you send email? If you have each process be responsible for sending a message, it might generate an error, begin sending email, generate a second error, begin sending email, and get in an endless loop. With the log approach you won't have to worry about that nasty corner case.

[john_oshea]

OK, it's not

(a) a direct answer to your question, or

(b) even remotely an all-Perl solution,

...but Nagios might be worth a look, especially as I see "network admin", "email" and "paging" all in the first paragraph. And Nagios is very very good at that kinda thing. And you get pretty web pages "for free" as well.

You can, it seems, write your plugins in Perl, so hooking up existing code may be less hassle than it might seem.

Of course if you're aware of all this already, I'll go off and soak my head :-)

[pileofrogs]

I'd never heard of Nagios before (I'm kindof just now learning to crawl out from under my rock), so thanks for the link. I've used mon, which is probably the ancient ancestor of Nagios.

I'm not so much looking to monitor services, as I want to keep tabs on all my scripts, hence the idea of a perl module. For instance, if the script that creates web accounts horks or even just has a few grumblings, I want an email about that.

[cees]

I'm not so much looking to monitor services, as I want to keep tabs on all my scripts

I would say that those are both the same thing. You want to make sure that if something fails you hear about it.

Nagios can work in active and passive mode. That means that it can actively check that services are running (like a webserver), or it can passively listen for problems (ie a script has a problem and sends a notification to nagios).

Also, Nagios handles all of the emailing/paging for you, and can control how many messages are sent, you can even configure it to send messages to a different group of people on the weekend, or at night. In other words, the problems you are trying to solve have mostly been solved for you already.

The part that you would need to write, is a simple interface module in perl that can sent alerts to nagios. That would be quite useful I think.

Even if you decide to write everything yourself, at least have a look at Nagios so that you can see the types of features can be useful in a monitoring system.

[wazzuteke]

I think both the comments above are great solutions to your quandary, along with being very similiar.

The Nagios software will conduct a number actions for you, however you specify. It will watch for normal activity, log in specified events and notify when necessary. You can also schedule outages along with other things relative to a centralized monitoring system. The one pretty big downfall is that it is not the easiest software to customize, particularly when you are looking for something used for simple scripts.

The second solution, in your case, is most likely ideal for you and is really a VERY simplified version of Nagios. In your case, along with the wonders of Perl, there are a number of ways to take care of this. I'll add a simple example that might hopefully spark your imagination.

First, hopefully you wont have to be weary of every line of code in your scripts. Generally speaking, there are a limited numbrer of things that will unexpectedly cause your script to fail. In these circumstances, you can 'catch' these by using the eval{} block, followed by a test for $@ special variable.

For example:


eval { 
  die 'Ack!';
};
print "ERROR: $@\n";

exit;
__END__

Will output: ERROR: Ack! Using this, you can:

eval {
  # Something that can die ...
};
Your::CustomMod::launch_error( $@ ) if ( $@ );

This way, you can programatically avoid areas where you want a die, warn, etc. to happen, yet catch the unexpected. You can be sure you catch only what you want to catch.
Good Luck

[adrianh]

You might want to look at Log::Log4perl. It can do everything you want with a little work:

read contact info from a central file:

Check. You can do something like Log::Log4perl::init('/etc/log4perl.conf'); at the start of your script to setup logging for all modules.

Control how often emails get sent

Write a custom appender and you can make it do whatever you like. Log::Log4perl::Appender::Buffer could probably do what you want.

Optionally override normal warn/die/carp/croak calls

See My program already uses warn() and die(). How can I switch to Log4perl? in the FAQ.

Optionally handle unexpected death as well as die/croak calls

See How can I make sure my application logs a message when it dies unexpectedly? in the FAQ.

Make it so the user can do as much configuration as possible at the beginning

You can do all the configuration in one call at the start.

[Perl Mouse]

It's not the way I would do it nowadays. Granted, long time ago, I build in paging/emailing into my applications, either directly or by using a module or library.

I no longer do. If an application has something to say, syslog is the way to go. The great thing about syslog is that it is centralized - this way I change who needs to be paged/emailed and with what frequency without having to reconfigure any application. I don't have to restart applications, and it's hard to forget one. From /etc/syslog.conf, I can decide which messages are to be send into a named pipe, and at the other end, I can have a program doing the messaging.

[pileofrogs]

Thanks to everyone for your comments.

• I took a look at Log::Log4Perl, and it looks very cool. I'll take a closer look this week.
• I like the idea of logging to syslog and then scanning through the logs, especially if I use a central logging server.
• Regarding the eval {} test idea, I actually do worry about every line of code causing my script to die. Or more accurately, the one time I forget the eval {} test is the one time I'm going to really eat my own foot. So, I really need something that logs/emails problems that I don't forsee.
• Nagios sounds like something nice for monitoring if your web server is running or something like that, but I really want something that's going to send me email when my account creation script finds a line from the student registrations dump that contains a field it can't parse. Am I missing some features of Nagios that are cooler than I realise?

Thanks!

[john_oshea]

Regarding the Nagios approach, you could have your program log errors to file, then use Nagios to monitor the file - possibly using the check_logs plugin

If you're concerned about your programs terminating unexpectedly, you could either:

• do some basic monitoring to see if the process is still running (for long-running processes), or
• log a 'program starting' message, then write a plugin to check for subsequent 'in progress' messages (or a corresponding 'program finishing' message) within a certain time period, and raise an alert if you've not heard anything.

That would be the stuff that's most applicable to your situation as I understand it - there are plugins for an awful lot of sysadmin-related tasks, so I guess your use of Nagios is going to be dictated by how much of the other functionality is going to be of use. If your sole use is going to be for monitoring your own code, installing and configuring Nagios is probably going to be overkill. Still, it's nice to have options, eh? ;-)