Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^2: Design advice: Classic boss/worker program memory consumption

by shadrack (Acolyte)
on May 22, 2014 at 02:36 UTC ( #1087045=note: print w/ replies, xml ) Need Help??


in reply to Re: Design advice: Classic boss/worker program memory consumption
in thread Design advice: Classic boss/worker program memory consumption

I thought of that, but I recall reading somewhere that if you allocate a big chunk of memory in perl and then release it (by way of undef'ing the variable or whatever) that memory is not returned to the operating system -- it's still owned by the perl interpreter as far as Linux is concerned. So you can write %dataset to disk and undef it, but w/rt children, forking, and copy-on-write, the operating system will treat that memory as if it's still in use. Thus, you gain nothing.


Comment on Re^2: Design advice: Classic boss/worker program memory consumption
Re^3: Design advice: Classic boss/worker program memory consumption
by flexvault (Parson) on May 22, 2014 at 11:20 UTC

    shadrack,

    Interesting, I just checked on AIX and the memory was freed for the child, but on Linux(Debian) the memory was still allocated for the child. So it depends on the OS.

    You could try this program on your system and see if the memory is freed or not? Note: My test results are at the bottom of the script.

    #!/usr/local/bin/perl use strict; use warnings; my $NAME = ""; my ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage +($$,$NAME,0); print "Before: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F +B_CacheMemoryRSS\n"; { my %RCache; for my $key ( 1_000 .. 11_000 ) { $RCache{$key} = chr( $key % 255 ) x 2_048; } ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag +e($$,$NAME,0); print "After: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory +: $FB_CacheMemoryRSS\n"; } ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage($$ +,$NAME,0); print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $FB_Ca +cheMemoryRSS\n\n"; my $fork = fork(); if ( $fork ) { ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag +e($$,$NAME,0); print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F +B_CacheMemoryRSS\n\n"; } else { ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usa +ge($$,$NAME,0); print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F +B_CacheMemoryRSS\n\n"; } exit; sub Display_Mem_Usage { # VSZ is size in KBytes of the virtual memory ( VSZ * 1024 ) # RSS is size in pages of real memory ( 1024 * RSS ) my $cpid = shift; my $name = shift; my $from = shift; my $var = ""; my $fh; my $arg = qq| -o "vsz rssize" -p $cpid|; open ( $fh, "-|", "/bin/ps $arg" ) or die "Display_Mem_Usage: No +t open \'$arg\': $!"; while (<$fh>) { $var .= $_; } close $fh; my $rno = my @ref = split(/\n/,$var); if ( $rno < 2 ) { return ( - +1, -1 ); } my $info = join(" ", split " ", $ref[1]); my ($vmem,$rmem) = ( split(/\ /,$info) ); return ( $vmem , $rmem ); } __END__ AIX:# perl ckmem.cgi Before: Virtual Memory: 776 Real Memory: 852 After: Virtual Memory: 21904 Real Memory: 21980 41030: Virtual Memory: 21908 Real Memory: 21984 40244: Virtual Memory: 424 Real Memory: 404 ## Child 41030: Virtual Memory: 21912 Real Memory: 21900 AIX:# Linux:# perl ckmem.cgi Before: Virtual Memory: 5288 Real Memory: 1928 After: Virtual Memory: 26136 Real Memory: 22804 26362: Virtual Memory: 26128 Real Memory: 22804 26362: Virtual Memory: 26128 Real Memory: 22808 26369: Virtual Memory: 26128 Real Memory: 21600 ## Child Linux:#

    Regards...Ed

    "Well done is better than well said." - Benjamin Franklin

      Well, I'm on Linux, so the results are no surprise:

      Before: Virtual Memory: 122236 Real Memory: 2416 After: Virtual Memory: 144476 Real Memory: 24640 29118: Virtual Memory: 144476 Real Memory: 24640 29118: Virtual Memory: 144476 Real Memory: 24644 29122: Virtual Memory: 144476 Real Memory: 23160

        shadrack,

        Sorry :-(

        Regards...Ed

        "Well done is better than well said." - Benjamin Franklin

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1087045]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2014-12-23 05:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (135 votes), past polls