Catching processes stuck in an apparent loop

[powerhouse]

I have built a system that has been in a live environment now for about 6 months, every now and then I notice the script gets hung up on the server, then once one does 4 to 8(+/- a few) more processes get hung up too and just sit there building the server load up, then more connections time out and it causes problems. I then have to go in there and manually kill each process that appears to be hung up. I know not all of them are because we have at any point in time around 40+ people on the website.

Is there a way for me to kill the process running if the script hangs for more than 10 seconds? Every process I run in the script should take no longer on the server than 0.4 seconds to 1 second.

Thanks for any direction you can point me...

[hipowls]

Use an alarm. If you need timeouts accurate to a millisecond then I'd suggest ualarm from Time::HiRes but that's probably overkill here.

[aquarium]

are you using mod_perl? OR spawning a perl for each CGI? resource contention issues in the script for file(s) etc? also, mod_perl or mod_cgi for apache you have to be careful to always re-init all variables and filehandles, as you can end up with values from a different/concurrent web session.
do you want to show us your code?

[powerhouse]

Not using mod_perl, just using Perl v5.8.8
Also using Apache::Session::MySql

So when I view 'top' from SSH I see something like this when I notice a bunch of hangs:


26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

# Added these to show what it looks like when there are issues:

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

26034 holytea   18   0 13544  10m 2096 S   16  0.5   0:00.49 web.cgi

#There are a bunch in here like that, of course all have their own pid
+, I also see a bunch of httpd processes.
10099 nobody    15   0  158m  17m 4640 S    3  0.9  33:22.20 httpd    
+                
26001 root      15   0  2172 1036  792 R    1  0.0   0:00.16 top      
+                
    1 root      15   0  2044  596  512 S    0  0.0   0:04.62 init     
+                
    2 root      RT   0     0    0    0 S    0  0.0   0:09.06 migration
+/0              
    3 root      36  19     0    0    0 S    0  0.0   0:00.21 ksoftirqd
+/0              
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/
+0               
    5 root      RT   0     0    0    0 S    0  0.0   0:08.04 migration
+/1              
    6 root      34  19     0    0    0 S    0  0.0   0:00.39 ksoftirqd
+/1              
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/
+1               
    8 root      RT   0     0    0    0 S    0  0.0   0:10.65 migration
+/2              
    9 root      34  19     0    0    0 S    0  0.0   0:00.17 ksoftirqd
+/2              
   10 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/
+2               
   11 root      RT   0     0    0    0 S    0  0.0   0:07.09 migration
+/3              
   12 root      35  19     0    0    0 S    0  0.0   0:00.24 ksoftirqd
+/3              
   13 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/
+3               
   14 root      10  -5     0    0    0 S    0  0.0   0:00.31 events/0



So what I need is for web.cgi and index.cgi to be monitored so that if they run too long they are automatically 'killed'. I know a kill daemon would be the way to go, but I have no idea how to go about it. I've been looking for a few weeks but cannot find anything definative.

[Ultra]

I think the right question is: why do those processes "hang", which is in fact "don't stop" :)