Category: |
Utility Scripts |
Author/Contact Info |
Azhrarn |
Description: |
After a day of trying to figure out why one of my web servers was locking up, I found that it was using a bit too much memory. But I had no idea how much, and Linux memory reporting is a bit arcane at best. Especially with something like apache + mod_perl/php using shared memory pools. So after some analysis, I came up with the included script.
It has the following output:
Total memory available: 3.21G
Total used by apache2 (451 instances): 3.80G
Total used by other processes: 0.12G
Average memory used per apache2 process: 8.63M
Recommended number of processes based on Average: 381
Needed memory for 500 processes based on Average: 4.21G
Max memory used for apache2 process: 17.61M
Recommended number of processes based on Max: 186
Needed memory for 500 processes based on Max: 8.60G
Mean plus two Standard Deviations (bulk of usage under max): 11.28M
Recommended number of processes based on Mean + 2*Stdev: 291
Needed memory for 500 processes based on Mean + 2*Stdev: 5.51G
At the time I ran it, I had 450 instances out of a configured max of 700 active. Come to find out I only had the ram to support about 380 of those at best. :(
Hopefully some other people find this useful, as it's pretty hard to get a straight answer on "How many processes can I support?" Or even quickly analyze how much memory a threaded process with shared memory is using. Note I don't bother to include the root apache process, or the negligible amount of base shared memory.
This can also be applied to other arbitrary processes by changing the name it scans for. It is also dependant on the "top" utility, as included in Debian. I'm not sure how it will act under other variations of top, although it should be fairly robust at pulling out the column names.
|
#!/usr/bin/perl
# Copyright Erik Jacobson - erik@underhanded.org
use warnings; use strict;
use Statistics::Descriptive;
my $webuser = 'www-data'; # The user your apache children run as. I i
+gnore the root process.
my $command = 'apache2'; # The name of your processes as they appear t
+o the "top" command.
my @top = `top -bn1`;
my %header;
my $mem = Statistics::Descriptive::Full->new();
my $othermem = Statistics::Descriptive::Full->new();
my $totalmem;
foreach (@top)
{
s/^\s+|\s+$//sg;
my @line = split(/\s+/, $_);
if(/mem:\s+(\S+)\s+total/i)
{
$totalmem = convtok($1);
}
if(defined $line[3] && exists $header{res} && $line[0] =~ /^\d
++$/)
{
next unless (defined $line[$header{user}] && defined $
+line[$header{res}]
&& defined $line[$header{shr}] && defined $lin
+e[$header{command}]);
if(($line[$header{user}] eq $webuser) && ($line[$heade
+r{command}] eq $command))
{
$mem->add_data(convtok($line[$header{res}]) -
+convtok($line[$header{shr}]));
} else {
$othermem->add_data(convtok($line[$header{res}
+]) - convtok($line[$header{shr}]));
}
} elsif (!exists $header{res})
{
my %tempheader;
for (0 .. $#line)
{
$tempheader{lc($line[$_])} = $_;
}
if(defined $tempheader{user} && defined $tempheader{re
+s}
&& defined $tempheader{shr} && defined $temphe
+ader{command})
{
%header = %tempheader;
}
}
}
printf "Total memory available: %.2fG\n", ($totalmem / 1024 / 1024);
printf "Total used by $command (%d instances): %.2fG\n", $mem->count()
+, ($mem->sum() / 1024 / 1024);
printf "Total used by other processes: %.2fG\n\n", ($othermem->sum() /
+ 1024 / 1024);
$totalmem -= $othermem->sum();
printf "Average memory used per $command process: %.2fM\n", ($mem->mea
+n() / 1024);
printf "Recommended number of processes based on Average: %d\n", ($tot
+almem / $mem->mean());
printf "Needed memory for 500 processes based on Average: %.2fG\n\n",
+($mem->mean() / 1024 / 1024 * 500);
printf "Max memory used for $command process: %.2fM\n", ($mem->max() /
+ 1024);
printf "Recommended number of processes based on Max: %d\n", ($totalme
+m / $mem->max());
printf "Needed memory for 500 processes based on Max: %.2fG\n\n", ($me
+m->max() / 1024 / 1024 * 500);
printf "Mean plus two Standard Deviations (bulk of usage under max): %
+.2fM\n", (($mem->mean() + $mem->standard_deviation() * 2) / 1024);
printf "Recommended number of processes based on Mean + 2*Stdev: %d\n"
+, ($totalmem / ($mem->mean() + $mem->standard_deviation() * 2));
printf "Needed memory for 500 processes based on Mean + 2*Stdev: %.2fG
+\n", (($mem->mean() + $mem->standard_deviation() * 2) / 1024 / 1024 *
+ 500);
sub convtok
{
my $num = shift;
if($num =~ /^\s*([0-9.]+)k\s*$/i)
{
$num = $1;
} elsif ($num =~ /^\s*([0-9.]+)m\s*$/i)
{
$num = $1 * 1024;
} elsif ($num =~ /^\s*([0-9.]+)g\s*$/i)
{
$num = $1 * 1024 * 1024;
}
return $num;
}
|
Re: Analyzing Apache memory usage
by zentara (Cardinal) on Jul 18, 2008 at 15:31 UTC
|
I couldn't tell by your code whether it updates constantly, but you may be interested in linux memory leak monitor. If you have an xserver running on the apache machine, it will open a little Tk window in the bottom right corner, and update it every second. That way you can watch it over the day, just with a glance. If you need to run commandline, you could run your script under POE or even Roll your own Event-loop where you can setup a Glib timer.
(Of course, you can also put it in a simple loop where you run it, then sleep 5) ;-) But feel free to modify my MeM script, and add all your info into the Tk Label.
| [reply] |
Re: Analyzing Apache memory usage
by Anonymous Monk on Sep 22, 2020 at 09:36 UTC
|
The script is a little old but still works. However, on boxes with very large amounts of RAM (100G+), top doesn't give an answer in K anymore and the script breaks.
I took out the if statement in the main loop that grabs the memory and replaced it with:
my $free= `free|fgrep Mem`;
if($free=~/Mem\s*:\s+(\S+)\s+/i)
{
$totalmem = convtok($1);
}
else
{
die("Can't find memory info\n");
}
just above the foreach and that worked for our 680GB ram monster. | [reply] [d/l] |
Re: Analyzing Apache memory usage
by Anonymous Monk on Oct 29, 2011 at 20:25 UTC
|
Thank-you. Looks like our Apache is loading far too much:
./apache.memory.analyse.pl
Total memory available: 0.49G
Total used by apache2 (3 instances): 0.73G
Total used by other processes: 0.44G
Average memory used per apache2 process: 248.45M
Recommended number of processes based on Average: 0
Needed memory for 500 processes based on Average: 121.31G
Max memory used for apache2 process: 258.87M
Recommended number of processes based on Max: 0
Needed memory for 500 processes based on Max: 126.40G
Mean plus two Standard Deviations (bulk of usage under max): 270.93M
Recommended number of processes based on Mean + 2*Stdev: 0
Needed memory for 500 processes based on Mean + 2*Stdev: 132.29G
| [reply] [d/l] |
Re: Analyzing Apache memory usage
by Anonymous Monk on Oct 29, 2011 at 19:55 UTC
|
Useful script. People who stumble upon this .pl, but do not know perl, will copy and paste this, end up with line breaks in the wrong place, and then never use the script ;) | [reply] |
|
They should press on the Download button to download a clean copy of the script.
| [reply] |
Re: Analyzing Apache memory usage
by Anonymous Monk on Oct 29, 2011 at 20:10 UTC
|
# ./apache.memory.analyse.pl
Total memory available: 0.49G
Total used by apache2 (0 instances): 0.00G
Total used by other processes: 0.00G
Average memory used per apache2 process: 0.00M
Illegal division by zero at ./apache.memory.analyse.pl line 61.
| [reply] |
|
This is a copy and paste error. Use the Download button to download a clean copy of the script.
| [reply] |
|
|