See lots of 'httpd' processes appear
I'm not sure what the usual ratio is between available system resources
and system resources needed to keep up with Perlmonks requests, but _if_
were to dip below a certain critical point, then the number of new
processes would grow faster than the old processes could finish. If
that were the case, the total number of running processes could be
expected to increase steadily, further dividing the system resources
(notably RAM) available to each, in a vicious cycle, which would explain
the extremeness of the symptoms you describe.
However, that leaves open the question of what happens to trigger
the event in the first place. If the available system resources were
just barely adequate for handling normal (or normal peak) traffic,
then a slightly-more-than-normal traffic spike could trigger it,
but it seems like if the system were that close to maxed out all
the time you'd probably already know it. Are there things users
can do that cause substantially more activity on the server than
a normal request? Too many Super Search queries at once, perhaps,
or something along those lines?
but 'top' isn't particularly flexible but is still the best tool I've found available on this system so far
My immediate thought here is to look for process-related stuff
on the CPAN, looking for something that doesn't just shell out to ps,
preferably something Unix-oriented and written in pure Perl. I don't
have much experience working with process tables, though, beyond
what can be done with ps and top. update: My second
thought is that I'm sure you're already aware some versions of top can
show considerably more columns than they do show by default. The version
I have here (on FreeBSD) is quite impoverished, but ISTR that the version
of top that I used on Mandrake 9 had rather a lot of optional columns
and a loose marble rolling around in the back of my head suggests it
_may_ (it's been several months...) have had an option for showing
the parent process. I mention this only on the off chance that you
haven't already checked for it. Hit ? in top to see a list.
Sanity? Oh, yeah, I've got all kinds of sanity. In fact, I've developed whole new kinds of sanity. Why, I've got so much sanity it's driving me crazy.