Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re: PL_strtab/SHAREKEYS and copy-on-write leak

by sundialsvc4 (Abbot)
on Jul 21, 2018 at 14:30 UTC ( #1219011=note: print w/replies, xml ) Need Help??


in reply to PL_strtab/SHAREKEYS and copy-on-write leak

Speaking once more to this topic this time, a little more tangentally I would make the point that the auto-garbage-collected memory models that we enjoy in interpreted languages such as (but not limited to) Perl, very convenient though they are, come at a price.   There are memory-reads and memory-writes constantly being made in support of this magic.   Therefore, the working-set size of any process is larger than you might expect it to be, since “merely referring-to” a datum often causes memory-writes and therefore touched-pages that you might not intuitively expect.   In the OP’s case, the working set of the child blossomed to 16 megabytes.   Certainly unexpected by the OP.   And yet there was indeed a reason for it.   Even if you were not thinking in terms of having modified anything, Perl did.

The shared-data store needs to be managed by a shared-data handling API such as one of the very-many that are available in CPAN.   (I only listed the first one that came to mind.)   The shared data structure should not exist in the memory-space that would be forked, but in a shared data segment or in the address-space of a single service process.   (In the former case) Perl will not garbage-collect it.   The plumbing and magic needed to do the trick should be concealed within the CPAN module that you select.   It should not be necessary to hand-create a custom XS-based solution to this familiar requirement, in most scenarios:   the work has already been well-done by others.

  • Comment on Re: PL_strtab/SHAREKEYS and copy-on-write leak

Replies are listed 'Best First'.
Re^2: PL_strtab/SHAREKEYS and copy-on-write leak
by sezal (Novice) on Jul 30, 2018 at 08:14 UTC

    sundialsvc4, in my code sample %h in parent get destroyed and not accessible in child. The only purpose of it is to preallocate more memory for PL_strtab and make copy-on-write issue more noticeable.

    The problem that PL_strtab is shared data structure (not %h). It's solely controlled by Perl and there is no way to control it or use IPC::Shareable or any other well-known for me CPAN modules.

    Real life example:
    1. In apache/mod_perl, Starman or any other prefork environment everybody tries to preload as much as possible modules in parent process. Right?
    2. If any of preloaded modules creates hash (even temporary) with big number of keys Perl silently allocates more and more memory for internal PL_strtab hash.
    3. PL_strtab silently get touched in children on any attempt to use hashes.
    4. Problem even worse, because huge percentage of modules we preload are CPAN modules -> there is no way to know which of them overuse hashes resulting in increased memory footprint of parent process.
    Thanks everybody for replies. I'm with Perl for over 15 years and PL_strtab issue was not obvious for me and I assume not obvious for majority of Perl developers. I expected and still expect to get suggestion/ideas that would help others as well. I definitely didn't expect that my first post to be considered "garbage".

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1219011]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (5)
As of 2019-12-08 10:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?