Re: Re: HTTP Daemonology

[clearcache]

Two thoughts here - not really related:

1) Checking the weblogs really doesn't do much to solve his problem, though. His problem is with CPU time being hogged by that one process. I don't think checking web logs is going to do much beyond telling him whether or not his machine has been hit with a high number of HTTP requests recently. Does that necessarily correlate with poor performance? In some cases, it seems so...but I'm not convinced that web hits alone are going to grind his sun box to a halt. It shouldn't - especially since it sounds like most of the pages are static!

I do like his solution of running top and scraping the output for process info, though.

2) One of the things I do to monitor one of my websites is run a simple perl script in cron using LWP and HTTP::Request modules. This way, you can make your own request to the site, check a the url for response time, and respond accordingly. Either restart the server automatically through that cron job, or, at the very least, fire off an email to you warning of the potential problems.

[idnopheq]

A non-perl ( yes, I know ) possibility is Sun's SymbEL release 3 out on http://www.sunfreeware.com, which our webmaster runs on his Solaris boxen. He did this due to a custom bit of Java that leaks bad. He is quite taken with it, tho I admit to not playing w/ it. YMMV.

For point 2, that's a beauty idea, even if you only use it to give you a heads up of impending doom!

I'd also, on the point one above, check the firewall and IDS logs to see if anything less-than-tasty is coming from outside.

Lastlly, check out Sun's Sun Performance and Tuning Techniques doc ( you may have to register w/ http://sunsolve.sun.com ). The techniques are pretty light ( I run them from time to time on my heavily utilized firewalls - no perl - w/ negligable additional load ).

As an aside, sysadmins who hang their hopes on patches and new releases exclusively w/o understanding what is *actually* wrong have, IME & IMHO, very short tenures in quality IT staffs. SysAdmins who can diagnose, provide evidence, and occasionally cruft a work-around, thrive - w/ little REM sleep, tho.

UPDATE: My sysadmin comments were directed toward jlongino's web admin, and not at jlongino. jlongino++ for taking this on.

HTH
--
idnopheq
Apply yourself to new problems without preparation, develop confidence in your ability to to meet situations as they arrise.

~ Shunryu Suzuki

[clearcache]

Re: weblog checking - yes, checking for nasties coming from the outside is important...and would make a nice addition to any periodic monitoring of a website. I need to add that to my monitoring program...

This only complements your note about avoiding impending doom. I'd rather get warned about impending doom 2 minutes before it happens...rather than finding out about it 10 minutes AFTER the moment of doom. I guess that's also a quality of a good sys admin.

(I'm not a sys admin, though. Just a programmer. But my company doesn't seem to see a reason to hire a full-time sys admin for our 6 sun boxen + > 300 users. Guess that's why I get very little REM sleep... ;)

[adamsj]

I agree that top is way useful, but be aware there's been a security problem reported (granted, nearly a year ago) on "systems that have top installed with set user or group permissions".

I saw it here--unfortunately, there weren't many details or any further references in this report, which I guess is a compliment to the reader's presumed research skills.

adamsj

They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen