Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Memory consumption of forks on exit

by egga (Monk)
on May 19, 2011 at 10:59 UTC ( #905667=perlquestion: print w/replies, xml ) Need Help??
egga has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks!

I have a problem and no idea how to solve it. I already spent some time googling and searching the monastery, but perhaps anyone can point me to the right node or site.

We have a script that builds up a large data structure and afterwards forks (we're on linux). The children do only read on that structure, so there is no need to copy it (which is important, because the machines don't have enough memory to handle a copy of the structure for each fork).

If we terminate the children using exit(), the structure is being copied for each child. If they get terminated using POSIX::_exit that will not happen. I think that is not nice and really would appreciate any other way to do that. Because I want an END block be called for each child and I don't have a clue, how to get this working.

If you wonder about that copy behavior, I prepared a small script, which proves the point (although i guess, most of you know it already). I watched my memory usage while execution and see a pretty high peak after the sleep if I alter the POSIX::_exit to a normal exit. Perhaps you need to adjust the array size for your computer...

#!/usr/bin/perl use strict; use warnings; use POSIX; my @array; for my $i (1 .. 100_000) { push @array, '-' x 1000; } print "before forking\n"; sleep 1; for (1 .. 10) { POSIX::_exit(0) unless fork; }

Replies are listed 'Best First'.
Re: Memory consumption of forks on exit
by salva (Abbot) on May 19, 2011 at 11:10 UTC
    That happens because when the interpreter shuts down, it deallocates all the Perl structures one by one, effectively writing to the memory where they are stored.

    Under Unix/Linux, that means that as the memory is write-accessed, the kernel has to perform a copy-on-write of the underlaying pages.

    As a workaround, try calling _exit or exec "/bin/true"; at the end of the END blocks.

      Ok, thats a good explanation. But I still want to execute the end blocks of the children, which doesn't happen if I try your workarounds.

        Then its simple, don't use end blocks, instead use
        used_to_be_an_end_block(); dance(); POSIX::_exit(0);
Re: Memory consumption of forks on exit
by sundialsvc4 (Abbot) on May 19, 2011 at 14:09 UTC

    A somewhat-difficult redesign of your strategy might prove necessary... or go to the hardware-store and buy lots more RAM.

      Ok, at least i found a solution without greater redesigns. Thanks for your advise fellow monks!

      Also nice to find out that those blocks work as subs, too. Thanks again.

      @sundialsvc4: Got any ideas how the strategy I laid out so far may be improved? Probably that's not gonna happen in the short term, but any alternatives would be appreciated.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://905667]
Approved by salva
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2017-12-13 23:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What programming language do you hate the most?




















    Results (381 votes). Check out past polls.

    Notices?