Memory consumption of forks on exit

egga has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks!

I have a problem and no idea how to solve it. I already spent some time googling and searching the monastery, but perhaps anyone can point me to the right node or site.

We have a script that builds up a large data structure and afterwards forks (we're on linux). The children do only read on that structure, so there is no need to copy it (which is important, because the machines don't have enough memory to handle a copy of the structure for each fork).

If we terminate the children using exit(), the structure is being copied for each child. If they get terminated using POSIX::_exit that will not happen. I think that is not nice and really would appreciate any other way to do that. Because I want an END block be called for each child and I don't have a clue, how to get this working.

If you wonder about that copy behavior, I prepared a small script, which proves the point (although i guess, most of you know it already). I watched my memory usage while execution and see a pretty high peak after the sleep if I alter the POSIX::_exit to a normal exit. Perhaps you need to adjust the array size for your computer...

#!/usr/bin/perl

use strict;
use warnings;

use POSIX;

my @array;
for my $i (1 .. 100_000) {
    push @array, '-' x 1000;
}

print "before forking\n";
sleep 1;

for (1 .. 10) {
    POSIX::_exit(0) unless fork;
}
[download]

Comment on Memory consumption of forks on exit Download Code

Replies are listed 'Best First'.
Re: Memory consumption of forks on exit by salva (Canon) on May 19, 2011 at 11:10 UTC
That happens because when the interpreter shuts down, it deallocates all the Perl structures one by one, effectively writing to the memory where they are stored. Under Unix/Linux, that means that as the memory is write-accessed, the kernel has to perform a copy-on-write of the underlaying pages. As a workaround, try calling `_exit` or `exec "/bin/true";` at the end of the `END` blocks.	[reply] [d/l] [select]
Re^2: Memory consumption of forks on exit by egga (Monk) on May 19, 2011 at 11:22 UTC
Ok, thats a good explanation. But I still want to execute the end blocks of the children, which doesn't happen if I try your workarounds.	[reply]
Re^3: Memory consumption of forks on exit by Anonymous Monk on May 19, 2011 at 11:36 UTC
Then its simple, don't use end blocks, instead use `used_to_be_an_end_block(); dance(); POSIX::_exit(0);` [download]	[reply] [d/l]
Re^4: Memory consumption of forks on exit by egga (Monk) on May 19, 2011 at 11:45 UTC
Re^5: Memory consumption of forks on exit by salva (Canon) on May 19, 2011 at 11:55 UTC
Some notes below your chosen depth have not been shown here
Re: Memory consumption of forks on exit by sundialsvc4 (Abbot) on May 19, 2011 at 14:09 UTC
A somewhat-difficult redesign of your strategy might prove necessary... or go to the hardware-store and buy lots more RAM.	[reply]
Re^2: Memory consumption of forks on exit by egga (Monk) on May 20, 2011 at 05:32 UTC
Ok, at least i found a solution without greater redesigns. Thanks for your advise fellow monks! Also nice to find out that those blocks work as subs, too. Thanks again. @sundialsvc4: Got any ideas how the strategy I laid out so far may be improved? Probably that's not gonna happen in the short term, but any alternatives would be appreciated.	[reply]


more useful options
	PerlMonks