Memory consumption of forks on exit

[egga]

Hi Monks!

I have a problem and no idea how to solve it. I already spent some time googling and searching the monastery, but perhaps anyone can point me to the right node or site.

We have a script that builds up a large data structure and afterwards forks (we're on linux). The children do only read on that structure, so there is no need to copy it (which is important, because the machines don't have enough memory to handle a copy of the structure for each fork).

If we terminate the children using exit(), the structure is being copied for each child. If they get terminated using POSIX::_exit that will not happen. I think that is not nice and really would appreciate any other way to do that. Because I want an END block be called for each child and I don't have a clue, how to get this working.

If you wonder about that copy behavior, I prepared a small script, which proves the point (although i guess, most of you know it already). I watched my memory usage while execution and see a pretty high peak after the sleep if I alter the POSIX::_exit to a normal exit. Perhaps you need to adjust the array size for your computer...

#!/usr/bin/perl

use strict;
use warnings;

use POSIX;

my @array;
for my $i (1 .. 100_000) {
    push @array, '-' x 1000;
}

print "before forking\n";
sleep 1;

for (1 .. 10) {
    POSIX::_exit(0) unless fork;
}

[salva]

That happens because when the interpreter shuts down, it deallocates all the Perl structures one by one, effectively writing to the memory where they are stored.

Under Unix/Linux, that means that as the memory is write-accessed, the kernel has to perform a copy-on-write of the underlaying pages.

As a workaround, try calling _exit or exec "/bin/true"; at the end of the END blocks.

[egga]

Ok, thats a good explanation. But I still want to execute the end blocks of the children, which doesn't happen if I try your workarounds.

[Anonymous Monk]

Then its simple, don't use end blocks, instead use

used_to_be_an_end_block();
dance();
POSIX::_exit(0);

[sundialsvc4]

A somewhat-difficult redesign of your strategy might prove necessary... or go to the hardware-store and buy lots more RAM.

[egga]

Ok, at least i found a solution without greater redesigns. Thanks for your advise fellow monks!

Also nice to find out that those blocks work as subs, too. Thanks again.

@sundialsvc4: Got any ideas how the strategy I laid out so far may be improved? Probably that's not gonna happen in the short term, but any alternatives would be appreciated.