Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Design advice: Classic boss/worker program memory consumption

by flexvault (Parson)
on May 21, 2014 at 15:22 UTC ( #1086994=note: print w/ replies, xml ) Need Help??


in reply to Design advice: Classic boss/worker program memory consumption

shadrack,

    "Does anybody see a solution that's better than either of the above in terms of SIMPLICITY and EFFICIENCY -- preferably one that doesn't require a major overhaul?"

After re-reading your post several times, it appears your problem is the expanding size of the '%dataset' hash. One solution is to save the assembled '%dataset' hash to disk and then clear the hash before forking the children.

As each child asks for more work, the 'boss' reads the disk for the next unit of work and passes that to the child for processing. You could do this with a database, but that just adds overhead that isn't needed. (Note: I'm assuming the hash(now file) is deleted after the boss completes the run.) Also, you don't say anything about the contents of the data, so you may want to include the size of each unit of work on disk, and read just that amount. Perl takes care of this for you in the hash.

This solution is very simple and would be easy to test/verify. Hope this helps!

Regards...Ed

"Well done is better than well said." - Benjamin Franklin


Comment on Re: Design advice: Classic boss/worker program memory consumption
Re^2: Design advice: Classic boss/worker program memory consumption
by shadrack (Acolyte) on May 22, 2014 at 02:36 UTC

    I thought of that, but I recall reading somewhere that if you allocate a big chunk of memory in perl and then release it (by way of undef'ing the variable or whatever) that memory is not returned to the operating system -- it's still owned by the perl interpreter as far as Linux is concerned. So you can write %dataset to disk and undef it, but w/rt children, forking, and copy-on-write, the operating system will treat that memory as if it's still in use. Thus, you gain nothing.

      shadrack,

      Interesting, I just checked on AIX and the memory was freed for the child, but on Linux(Debian) the memory was still allocated for the child. So it depends on the OS.

      You could try this program on your system and see if the memory is freed or not? Note: My test results are at the bottom of the script.

      #!/usr/local/bin/perl use strict; use warnings; my $NAME = ""; my ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage +($$,$NAME,0); print "Before: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F +B_CacheMemoryRSS\n"; { my %RCache; for my $key ( 1_000 .. 11_000 ) { $RCache{$key} = chr( $key % 255 ) x 2_048; } ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag +e($$,$NAME,0); print "After: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory +: $FB_CacheMemoryRSS\n"; } ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage($$ +,$NAME,0); print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $FB_Ca +cheMemoryRSS\n\n"; my $fork = fork(); if ( $fork ) { ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag +e($$,$NAME,0); print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F +B_CacheMemoryRSS\n\n"; } else { ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usa +ge($$,$NAME,0); print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F +B_CacheMemoryRSS\n\n"; } exit; sub Display_Mem_Usage { # VSZ is size in KBytes of the virtual memory ( VSZ * 1024 ) # RSS is size in pages of real memory ( 1024 * RSS ) my $cpid = shift; my $name = shift; my $from = shift; my $var = ""; my $fh; my $arg = qq| -o "vsz rssize" -p $cpid|; open ( $fh, "-|", "/bin/ps $arg" ) or die "Display_Mem_Usage: No +t open \'$arg\': $!"; while (<$fh>) { $var .= $_; } close $fh; my $rno = my @ref = split(/\n/,$var); if ( $rno < 2 ) { return ( - +1, -1 ); } my $info = join(" ", split " ", $ref[1]); my ($vmem,$rmem) = ( split(/\ /,$info) ); return ( $vmem , $rmem ); } __END__ AIX:# perl ckmem.cgi Before: Virtual Memory: 776 Real Memory: 852 After: Virtual Memory: 21904 Real Memory: 21980 41030: Virtual Memory: 21908 Real Memory: 21984 40244: Virtual Memory: 424 Real Memory: 404 ## Child 41030: Virtual Memory: 21912 Real Memory: 21900 AIX:# Linux:# perl ckmem.cgi Before: Virtual Memory: 5288 Real Memory: 1928 After: Virtual Memory: 26136 Real Memory: 22804 26362: Virtual Memory: 26128 Real Memory: 22804 26362: Virtual Memory: 26128 Real Memory: 22808 26369: Virtual Memory: 26128 Real Memory: 21600 ## Child Linux:#

      Regards...Ed

      "Well done is better than well said." - Benjamin Franklin

        Well, I'm on Linux, so the results are no surprise:

        Before: Virtual Memory: 122236 Real Memory: 2416 After: Virtual Memory: 144476 Real Memory: 24640 29118: Virtual Memory: 144476 Real Memory: 24640 29118: Virtual Memory: 144476 Real Memory: 24644 29122: Virtual Memory: 144476 Real Memory: 23160

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1086994]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2014-09-21 12:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (168 votes), past polls