Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re: PL_strtab/SHAREKEYS and copy-on-write leak

by sundialsvc4 (Abbot)
on Jul 21, 2018 at 02:45 UTC ( #1218980=note: print w/replies, xml ) Need Help??

in reply to PL_strtab/SHAREKEYS and copy-on-write leak

I would argue that this behavior is not a leak ... that it is exactly the behavior you would logically expect, and it is necessary and correct.   (But IMHO it is not the right way to do what you apparently are wanting to do.)

If you fork a process that owns a hash with two million keys in it, then copy-on-write semantics allow the forking to occur quickly.   But as each value is modified by the child process, the operating system makes a duplicate copy of the entire virtual-memory page.   In the example that you describe, this occurs for every entry because you touched it.   (Even if you assigned a variable or bucket the same value that it already had.)   16 megabytes of data are duplicated because they had to be.

And you don’t even have to touch it.   For instance:   my $foo = $shared_hash->{'bar'};   Like it or not, that's a new reference to that hash-element, and so its reference count just went up.   (And when $foo goes out of scope, it will go down.)   Since it just went up, a write and therefore a copy-on-write just occurred.   (I presume that this is intended to be a read-only data structure ...)   Even if you merely intend to reference it, you’re fairly doomed to copy-on-write it, if you try to use ordinary Perl constructs.   You need to take a fundamentally different approach, and (of course) CPAN has a number of ready-made alternatives to choose from.

I submit that, if you need to manipulate a large shared data structure, you need to resort to something like, say, IPC::Shareable to store the data in a properly-protected shared memory segment.   Or you might use interprocess communication along the same lines as the memcached daemon in Unix/Linux.   The large-data-structure should not be part of what is affected by fork.

  • Comment on Re: PL_strtab/SHAREKEYS and copy-on-write leak

Replies are listed 'Best First'.
    A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1218980]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (8)
As of 2019-12-14 00:01 GMT
Find Nodes?
    Voting Booth?

    No recent polls found