Re: Auto-restarting script if it dies

[merlyn]

My DBI Logger helper script needs to be running all the time, so I wrote a watchdog that sits above it. If the heavy lifting child ever exits, the parent restarts it immediately. And since the parent doesn't do much besides just sit there, it probably won't go away.

[sgifford]

On Unix, daemontools provides a fairly straightforward way to do exactly what merlyn describes without writing any code. It goes a step further, and has one additional process monitoring all of the monitoring process, with that process restarted by init. That's pretty foolproof, and it's straightforward to set up many monitored services.

[mojodaddy]

As a matter of fact, I spent a few hours grappling with daemontools, and it's a perfect example of the freakishly opaque "explanations" I described in my original post. Here's its FAQ entry for the question "How do I create a service directory?":

Answer: The only required component of your service directory is an executable file, ./run, that runs your daemon in the foreground, exiting when your daemon exits.

Huh? Does that mean I'm supposed to create ./run? And if so, what is it? And how, exactly, do I create it? The FAQ says:

Typically ./run is a shell script. For example:

#!/bin/sh
echo starting
exec clockspeed

"Typically?" So is my situation typical? How can I tell? And if ./run is a shell script in a typical case, what is it in other cases?

So let's assume that I am supposed to write a shell script (which in my case would execute my perl script?) and save it as ./run in my service directory. But if so, why not just say so? Plus, would it kill them to explain how? It's like if I said, "How is a moon-rock different from a regular rock?" and they said "OK, first go to the moon and get some rocks, and I'll tell you the rest when you get back..."

Of course the reality is that if the daemontools FAQ had been anything less than completely baffling, I wouldn't have written my solution, in which case I would never had received a personal response from merlyn! So it's all good. : )

[jethro]

'typically' doesn't refer to the situation but to the script, which could be a c-program, a shell script or a perl-script. It's left up to you what you want in it. You could put the deamon itself there or just a script that starts the daemon, or a script that starts a script that starts the daemon ;-)

In most cases you've got the daemon executable somewhere else, so typically aka usually you put a script there.

[sgifford]

The key to understanding these tools is to read the documentation for the programs to see exactly what they do, then extrapolate from there. See, for example, the supervise manual. The reason the instructions are a bit vague is because, as with most of Unix, the software consists of a small tool which can be combined with other small tools in any number of different ways; how you will combine them will depend on exactly what you hope to accomplish.

In this case, the key thing that supervise does is execute whatever's in the file named run, then starts it back up again if it exits. run can be a shell script, a perl script, a binary program, a symlink, or anything else that Unix knows how to execute. In your case, you probably want either a shell script to exec your program, or else a symlink to your actual program as your run script.

[mojodaddy]

Wow... You'll have to forgive me for being a little star-struck here. I'm both hideously embarrassed and oddly proud to know that you've read and responded to my humble code. (But mostly the former.) Thanks, merlyn.

[ant]

This could be just me, but it seems a bit odd to start one script, and then start another script to check the first script.

Perhaps you could in the checker.pl delete the checkme.txt file (under the line where you declare var $checkme, that way, on start up it's deleted and does not pick up an old version of the file with an old temp file name, as checkme.txt is never deleted any where in your code)

Run the checker.pl script, first time in the loop, file does not exist, starts always_running.pl and creates checkme.txt and then the checker script continues checking checkme.txt.

Just trying to refine the process, I liked the idea lots.

Ant

[mojodaddy]

No, checkme.txt is never deleted; it gets overwritten each time the script starts, though, so the scenario you described never occurs.

The thing's been working great, too -- almost too well! It was pretty hilarious the first couple of times I tried to restart the script because I'd made a change to the code. It just kept popping back up again! It was like playing whack-a-mole. I've since concluded that stopping the checker first and then stopping the script is the preferred technique!