OS memory reclamation with threads on linux

[zentara]

On 1 of the newsgroups, someone asked about the pros/cons differences between using forks or threads. Well the newsgroup is c oriented, and I thought I would mention the way that interpreted languages (like Perl) will hold onto memory released by threads. Specifically I mentioned that if a high memory usage thread is run first, then a series of low memory usage threads, the memory of the process will stay at the high memory usage. I used the following script to demonstrate. (I have a 1 gig ram on a linux 2.6.22 kernel). The results were unpredictable, but generally followed the rule that heavy mem use is freed, medium use is half freed, and light use ( <5%) is not freed( but may be on occaision).

#!/usr/bin/perl
use warnings;
use strict;
use threads;

my $count = 0;

#  load a mem intensive thread first, then watch how mem  drops
# notice the difference in memory reclamation( top) after
# the following thread is run with heavy,medium, light mem

# on my machine
# heavy mem use will skyrocket to 75%, then drop to 2%
# medium will jump to 40% then drop to 20%
# light use will jump to 5% then stay there
# none use will stay at .3 %

#my $t = threads->new(\&do_my_thing, $count, 1000000); #light
#my $t = threads->new(\&do_my_thing, $count, 4000000); #medium
my $t = threads->new(\&do_my_thing, $count,    8000000);  #heavy
#my $t = threads->new(\&do_my_thing, $count, 1);  #none

$t->join;
select(undef,undef,undef,.1);
    
while (1) {
        $count++;
        print "thread $count started\n";         
        my $t = threads->new(\&do_my_thing, $count, 1 );
        $t->join;
        select(undef,undef,undef,.1);
    }

<>; #wait for keypress to end, to check top

sub do_my_thing {
    my ($val,$cdown) = @_;
    my %hash;
    for(0..$cdown){ $hash{$_} = 'a'; }
        
    select(undef,undef,undef,.1);
    print "thread $val ended\n";         
}

So it seems that the kernel is deciding how much to free, depending on some internal formula.

The question is: Can we trust top/ps to really report what is going on. Since most threaded scripts may go 10% up occaisionally, if they don't reach the kernel threshold for freeing, is that dead memory, or will the kernel auto-free more (the diff between 5% and .3%) if it needs the memory for other apps.

This would be great news for threaded apps, but I worry about not being able to trust top/ps, or the kernel doing things behind the scenes.

Anyways, this may be a mechanism to reclaim memory in Perl. I would appreciate any thoughts, comments on what is happening.

[chromatic]

Can we trust top/ps to really report what is going on.

Not usually; Linux allows memory overcommit. As I understand it (and simplifying somewhat), any application can request as much memory as is present on the machine (the VIRT column in top), but only the RES column is actually in use at any time. If all of the RES memory exceeds physical memory, it's swap time.

Without knowing what you watched grow and shrink in top, it's difficult to explain what happened.

[zentara]

I watch the %MEM columns in top/ps. It's been confusing testing, because when there is a small percent difference between the peak and normal mem usage, sometimes the kernel shows %MEM drop, other times not. If the diff is big, the kernel immediately returns it.

My real question is what are people talking about when they claim their threaded apps are increasing in memory. Do I just say "forget it.... the kernel will grab it if needed", or "you have a thread design problem"

I'm also wondering if this has different behavior with the older 5.8 threads vs. the latest threads (which seem to run better), and/or kernel versions.

It seemed like just last week that this was a constant problem with Perl threads, and now suddenly it seems like a great thing, where you can reclaim memory from mem-intensive Perl scripts, by putting the mem-intensive stuff into a thread.

[jbert]

I watch the %MEM columns in top/ps

Well, that's proportional to RES, it's the amount of physical memory used by the app at that moment (including all loaded libs and possibly shared memory). It only reflects your apps "memory usage" if the box isn't under memory pressure (i.e. too much running at once). If your box *is* under memory pressure, then RES and %MEM will relate to what the OS is running at that moment - that figure will go down when bits of your app is paged out for something else to run, even though your app is still "using the same amount of memory" (i.e. it hasn't free'd anything).

My real question is what are people talking about when they claim their threaded apps are increasing in memory.

My first guess would be that they've just got a good old-fashioned memory leak.

The OS generally makes good decisions regarding memory use. The best advice is to not worry about it too much until you have a problem app, unless you're into learning things (which is cool).

[perrin]

Perl's memory usage with threads is fairly unique, I think. It copies almost everything when you start a new thread. This is why forking uses much less memory than threads on OSes like Linux with good copy-on-write support.