Re: Design advice: Classic boss/worker program memory consumption

[flexvault]

shadrack,

"Does anybody see a solution that's better than either of the above in terms of SIMPLICITY and EFFICIENCY -- preferably one that doesn't require a major overhaul?"

After re-reading your post several times, it appears your problem is the expanding size of the '%dataset' hash. One solution is to save the assembled '%dataset' hash to disk and then clear the hash before forking the children.

As each child asks for more work, the 'boss' reads the disk for the next unit of work and passes that to the child for processing. You could do this with a database, but that just adds overhead that isn't needed. (Note: I'm assuming the hash(now file) is deleted after the boss completes the run.) Also, you don't say anything about the contents of the data, so you may want to include the size of each unit of work on disk, and read just that amount. Perl takes care of this for you in the hash.

This solution is very simple and would be easy to test/verify. Hope this helps!

Regards...Ed

[shadrack]

I thought of that, but I recall reading somewhere that if you allocate a big chunk of memory in perl and then release it (by way of undef'ing the variable or whatever) that memory is not returned to the operating system -- it's still owned by the perl interpreter as far as Linux is concerned. So you can write %dataset to disk and undef it, but w/rt children, forking, and copy-on-write, the operating system will treat that memory as if it's still in use. Thus, you gain nothing.

[flexvault]

shadrack,

Interesting, I just checked on AIX and the memory was freed for the child, but on Linux(Debian) the memory was still allocated for the child. So it depends on the OS.

You could try this program on your system and see if the memory is freed or not? Note: My test results are at the bottom of the script.

#!/usr/local/bin/perl

    use strict;
    use warnings;

    my $NAME = "";

    my ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage
+($$,$NAME,0);
    print "Before: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F
+B_CacheMemoryRSS\n";
    {    my %RCache;
        for my $key ( 1_000 .. 11_000 )
        {    $RCache{$key} = chr( $key % 255 ) x 2_048;
        }
        ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag
+e($$,$NAME,0);
        print "After:  Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory
+: $FB_CacheMemoryRSS\n";
    }
    ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage($$
+,$NAME,0);
    print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $FB_Ca
+cheMemoryRSS\n\n";

    my $fork = fork();
    if ( $fork )
    {
        ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag
+e($$,$NAME,0);
        print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F
+B_CacheMemoryRSS\n\n";
    }
    else
    {    ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usa
+ge($$,$NAME,0);
        print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F
+B_CacheMemoryRSS\n\n";
    }

    exit;

sub Display_Mem_Usage
{    # VSZ is size in KBytes of the virtual memory ( VSZ * 1024 )
    # RSS is size in pages of real memory ( 1024 * RSS )
    my $cpid = shift; my $name = shift; my $from = shift;
    my $var = ""; my $fh;

    my $arg = qq| -o "vsz rssize" -p $cpid|;
      open ( $fh, "-|", "/bin/ps $arg" ) or die "Display_Mem_Usage: No
+t open \'$arg\': $!";
    while (<$fh>)
    { $var .= $_; }
    close $fh;
    my $rno = my @ref = split(/\n/,$var); if ( $rno < 2 ) { return ( -
+1, -1 ); }
    my $info = join(" ", split " ", $ref[1]);
    my ($vmem,$rmem) = ( split(/\ /,$info) );
    return ( $vmem , $rmem );
 }


__END__

   AIX:#  perl ckmem.cgi    
   Before: Virtual Memory: 776     Real Memory: 852
   After:  Virtual Memory: 21904   Real Memory: 21980
   41030: Virtual Memory: 21908    Real Memory: 21984
   
   40244: Virtual Memory: 424      Real Memory: 404   ## Child
   
   41030: Virtual Memory: 21912    Real Memory: 21900
   
   AIX:# 
   
   
   Linux:# perl ckmem.cgi
   Before: Virtual Memory: 5288    Real Memory: 1928
   After:  Virtual Memory: 26136   Real Memory: 22804
   26362: Virtual Memory: 26128    Real Memory: 22804
   
   26362: Virtual Memory: 26128    Real Memory: 22808
   
   26369: Virtual Memory: 26128    Real Memory: 21600   ## Child
   
   Linux:#

Regards...Ed

[shadrack]

Well, I'm on Linux, so the results are no surprise:

Before: Virtual Memory: 122236  Real Memory: 2416
After:  Virtual Memory: 144476  Real Memory: 24640
29118: Virtual Memory: 144476   Real Memory: 24640
29118: Virtual Memory: 144476   Real Memory: 24644
29122: Virtual Memory: 144476   Real Memory: 23160