Re^4: leak detection in other (non-perl) processes

[apprentice]

OK, let's say that I want to monitor process X, which could be affected by repeatedly killing process Y. That is, I am attempting to 'induce' leaks.

So, I get a baseline, like say '17' open file descriptors (FDs) for process X.

Then I loop for 'n' iterations, killing process Y, waiting a bit, getting current FD count, storing it, next.

Now, I may have a list that looks like one of the following:

For example, lets use file descriptors, here are some lists of possibl
+e values
A = 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 
+17 17 
B = 17 17 17 17 17 17 17 17 17 17 17 17 16 17 17 17 17 17 17 17 17 17 
+17 17 
C = 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 
+17 25 
D = 17 20 17 20 17 20 17 20 17 20 17 20 17 20 17 20 17 20 17 20 17 20 
+17 20 
E = 17 20 34 29 36 22 18 25 19 19 19 43 52 37 28 19 17 17 17 20 17 27 
+36 47 
F = 17 20 23 26 29 32 35 38 41 44 47 50 53 56 59 62 65 68 71 74 77 80 
+83 86
G = 17 20 20 26 25 26 30 34 38 41 44 39 42 46 50 58 52 55 55 55 55 56 
+59 58

Here are some of the thoughts I have jotted down:
Example A: not a leak, easy to determine, never changes

Example B: not a leak, almost never changes, value only 
ips for a moment, may reset low value, but does not reset 
average value (int), so does not represent a leak, easy 
to determine

Example C: does not appear to be a leak, almost never 
changes, only appears at end of data, low enough value 
should not change average value much, would statistics
help here, to show it as an outlier? may be able to use 
position as an indicator, no change ever until final 
position out of n positions would show it to be an outlier
(however, a single change earlier that did not reset could 
be a very slow leak, potentially, requiring more time to 
evaluate)
NOTE: a single change anywhere, that resets, is not a leak

Example D: not a leak, changes regularly, up and down the 
same amounts, harder to detect, can notice that the current 
number repeatedly drops to the low number (or thereabouts) 
and the current number only climbs to the max number or 
thereabouts (can keep average low and average max as well 
as running average), maybe keep counts of individual 
numbers or 'buckets' as well, but still possible to 
determine
NOTE: Related, but IS a leak of some sort, would be when it 
always drops back to the low or thereabouts, but always
or evetually always climbs to a higher max 

Example E: not a leak, changes regularly, but by varying 
amounts, ends on a high note, so would have to be mindful 
that it drops back down to or near the low frequently 
(frequency/counts) and recently (positional)

Example F: IS a leak, changes regularly, always up, but by 
varying amounts, easy to detect (can also count increases, 
decreases if it helps the other cases)

Example G: IS a leak, changes regularly, not every time, 
sometimes decreases, but relatively few decreases, 
continued upward trend 

NOTE: Should continually check that the PID of each process 
does not change, else the values are no longer any 
good!!!in which case, they will almost certainly reset to 
lower values, perhaps climbing again

Other considerations: Want to consider the slope, 
percentage-wise as well as absolute...???

[BrowserUk]

Maybe something like this will get you started. Try playing with the threshhold and moving average period (see examples):

#! perl -slw
use strict;
use List::Util qw[ sum min max reduce ];

my %cases = (
    A => [ qw[ 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 1
[phone] ] ],
    B => [ qw[ 17 17 17 17 17 17 17 17 17 17 17 17 16 17 17 17 17 17 1
[phone] ] ],
    C => [ qw[ 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 1
[phone] ] ],
    D => [ qw[ 17 20 17 20 17 20 17 20 17 20 17 20 17 20 17 20 17 20 1
[phone] ] ],
    E => [ qw[ 17 20 34 29 36 22 18 25 19 19 19 43 52 37 28 19 17 17 1
[phone] ] ],
    F => [ qw[ 17 20 23 26 29 32 35 38 41 44 47 50 53 56 59 62 65 68 7
[phone] ] ],
    G => [ qw[ 17 20 20 26 25 26 30 34 38 41 44 39 42 46 50 58 52 55 5
[phone] ] ],
);

our $V //= 0;
our $THRESHHOLD //= 1.1;
our $MOVING //= 5;

for my $case ( sort keys %cases ) {
    my $v = $cases{ $case };
    my @moving = map{
        sprintf "%.2f", sum( @{ $v }[ $_ .. $_+$MOVING  ] ) / $MOVING
    } 0 .. $#$v - $MOVING;

    my $min = min @moving;
    my $max = max @moving;
    my $leak = 'no leak';
    if(
        $moving[ 0 ] == $min and $moving[ -1 ] == $max
        and ( $min * $THRESHHOLD ) < $max
    ) {
        $leak = 'a leak ';
    }
    printf "Case $case has %s %s\n", $leak, $V ? "[ @moving ]" : '';
}

__END__
[20:34:52.28] C:\test>junk47 -THRESHHOLD=1.10 -MOVING=5
Case A has no leak
Case B has no leak
Case C has no leak
Case D has no leak
Case E has no leak
Case F has a leak
Case G has a leak

[20:35:07.79] C:\test>junk47 -THRESHHOLD=1.07 -MOVING=5
Case A has no leak
Case B has no leak
Case C has a leak
Case D has no leak
Case E has no leak
Case F has a leak
Case G has a leak

[20:35:16.82] C:\test>junk47 -THRESHHOLD=1.07 -MOVING=15
Case A has no leak
Case B has no leak
Case C has no leak
Case D has no leak
Case E has no leak
Case F has a leak
Case G has a leak

[20:35:25.46] C:\test>junk47 -THRESHHOLD=1.03 -MOVING=15
Case A has no leak
Case B has no leak
Case C has no leak
Case D has no leak
Case E has no leak
Case F has a leak
Case G has a leak

[20:35:32.66] C:\test>junk47 -THRESHHOLD=1.025 -MOVING=15
Case A has no leak
Case B has no leak
Case C has a leak
Case D has no leak
Case E has no leak
Case F has a leak
Case G has a leak

[Eliya]

As long as you don't set a fixed time frame for the observation/analysis, everything will be just a more or less useful heuristic.

Anyhow, I think you want to fit a trend line through the entire data¹, and if it isn't flat, you likely have a leak.  This assumes the evaluation interval is (considerably) longer than the period of any cyclic behavior you might observe.

___

¹ if you're worried about outliers, you might want to look for a statistics package that can compute the fit based on minimizing linear deviations, instead of squared deviations, as done in a standard "least squares" fit (see also LAD).