Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re^2: Design advice: Classic boss/worker program memory consumption

by shadrack (Acolyte)
on May 20, 2014 at 20:01 UTC ( #1086854=note: print w/replies, xml ) Need Help??

in reply to Re: Design advice: Classic boss/worker program memory consumption
in thread Design advice: Classic boss/worker program memory consumption

When spawning a worker, the boss does a fork, immediately followed by an exec, which is much more memory friendly (on modern Linux/Unix systems, copy-on-write means you need barely extra memory for a fork when neither of the programs modify the data).

copy-on-write does indeed make a fork more efficient, but you lose all that efficiency the minute you do an exec because the exec'd process starts over from scratch, loading its own separate copy of the perl interpreter, its own separate copy of the script, all the modules, etc. -- all taking up memory which would otherwise have been shared with the parent. This makes it LESS memory-friendly than just forking.

  • Comment on Re^2: Design advice: Classic boss/worker program memory consumption

Replies are listed 'Best First'.
Re^3: Design advice: Classic boss/worker program memory consumption
by moritz (Cardinal) on May 21, 2014 at 05:47 UTC

    You also lose all that efficiency when you start to read a lot of data structures in perl, because reading data ca n increase the reference counters, thus a write actually happens in the background, causing the operating system to copy the pages.

    So, don't assume stuff is memory unfriendly. Measure it. In your context, where it matters.

    Also, starting a new perl interpreter takes about 0.017s "cold" on my machine, and 0.002s when cached. Compared to a 1 minute run time of a worker thread, that's like 0.03% (cold) or 0.003% (warm) - nothing to worry about.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1086854]
[holli]: you know, monk days are fun and all. but too many of them are bad for your health
[1nickt]: ...modern car. Suggests using TLA+ or other software modeling language to autogenerate code.
1nickt is not sure more and more layers separating the programmer from the program is the right way to go...
[holli]: happy one, shmem ;)
[1nickt]: I had an interview couople of weeks ago, heard from a principal engineer that a friend of his who works at The Google told him that the self-teaching software that determines the dynamic pricing for their adverts...
[1nickt]: ... has become so complex that there is no one at the Goole who knows what it does.
[marto]: William Gibson was right
[moritz]: a classical problem with machine learning
[1nickt]: moritz isn;t the difference today that the machine learning algos are in charge of most stuffs?
[moritz]: depends on what you mean by "most stuff"

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (13)
As of 2017-11-21 13:02 GMT
Find Nodes?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:

    Results (301 votes). Check out past polls.