Re^2: Finding memory usage of perl structure efficiently

[dpavlin]

Thanks for lot for a pointer to newer version. On first run, it reported

Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814

Fiar enough, but overall it's fastest method to find size of perl structure:

# 20000 elements
            Rate size_0.71    reopen       cat size_0.72    freeze
size_0.71 80.3/s        --      -43%      -65%      -83%      -84%
reopen     142/s       77%        --      -38%      -71%      -71%
cat        227/s      183%       60%        --      -53%      -54%
size_0.72  485/s      505%      242%      114%        --       -1%
freeze     490/s      511%      245%      116%        1%        --
# 30000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 37.1/s        --      -54%      -65%      -83%      -84%
reopen    80.4/s      117%        --      -25%      -63%      -66%
cat        107/s      189%       33%        --      -51%      -54%
freeze     217/s      486%      170%      103%        --       -7%
size_0.72  234/s      530%      191%      118%        7%        --
# 40000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 22.1/s        --      -59%      -67%      -83%      -85%
reopen    54.1/s      144%        --      -20%      -58%      -64%
cat       67.4/s      204%       25%        --      -48%      -55%
freeze     130/s      487%      141%       93%        --      -13%
size_0.72  149/s      571%      175%      121%       14%        --
# 50000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 13.6/s        --      -63%      -70%      -81%      -85%
reopen    36.5/s      169%        --      -20%      -48%      -61%
cat       45.8/s      237%       25%        --      -35%      -51%
freeze    70.2/s      417%       92%       53%        --      -24%
size_0.72 92.6/s      582%      153%      102%       32%        --
# 60000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 9.97/s        --      -66%      -69%      -80%      -86%
reopen    28.9/s      190%        --      -10%      -43%      -59%
cat       31.9/s      220%       11%        --      -37%      -55%
freeze    50.5/s      407%       75%       58%        --      -28%
size_0.72 70.4/s      606%      144%      120%       39%        --
# 70000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 7.91/s        --      -65%      -69%      -79%      -86%
reopen    22.7/s      187%        --      -12%      -41%      -59%
cat       25.9/s      227%       14%        --      -32%      -54%
freeze    38.2/s      383%       68%       48%        --      -32%
size_0.72 56.1/s      609%      147%      117%       47%        --
# 80000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 5.78/s        --      -69%      -72%      -80%      -86%
reopen    18.9/s      227%        --      -10%      -35%      -55%
cat       21.0/s      264%       11%        --      -28%      -50%
freeze    29.1/s      404%       54%       39%        --      -31%
size_0.72 42.3/s      632%      124%      101%       45%        --
# 90000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 4.95/s        --      -69%      -72%      -80%      -86%
reopen    16.1/s      226%        --       -9%      -35%      -55%
cat       17.7/s      257%        9%        --      -29%      -51%
freeze    24.7/s      399%       53%       40%        --      -32%
size_0.72 36.1/s      630%      124%      105%       46%        --
# 100000 elements
            Rate size_0.71    reopen       cat    freeze size_0.72
size_0.71 4.21/s        --      -69%      -72%      -80%      -86%
reopen    13.8/s      227%        --       -9%      -35%      -56%
cat       15.2/s      261%       10%        --      -29%      -51%
freeze    21.3/s      407%       55%       40%        --      -31%
size_0.72 31.0/s      637%      125%      104%       45%        --

I would love to see version 0.71 depreciated :-)

[BrowserUk]

On first run, it reported Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814

I assume that you are running a 64-bit platform with 8GB of ram?

My version requires a small, static area of memory for tracking purposes. This has to be sized to cater for the size of the memory of the machine. It ships with this sized to cater for the maximum memory on a 32-bit machine (8192 * sizeof(void *).

On a 64-bit machine that has more than 4GB, it requires this be increased.

[dpavlin]

I'm using 64-bit kernel with 32-bit userland. It might make sense to bump it up to 16384 so 64-bit users won't have to touch source at expense of small memory increase.

[BrowserUk]

A year and a half ago when I release 0.72, the preponderance of people were running 32-bit and 8192 was already double what they required (most 32-bit processes are limited to 2GB and 8192 allows for 4GB). And even most 64-bit systems maxed at 4GB.

But, Moore's law moves inexorably on, and 8GB is probably about right to cover the next couple of years or so. If I decide to release a forked version, I will increase the default to 16384. I have been waiting for the appearance of *a* bug against the current version, but none has emerged..

There is a bug report, raised against 0.71, that also affects 0.72, but I consider that a bug (or rather, a regression), in the perl core. This because the the "bug" did not happen prior to the 5.10 release; and because it affects (and afflicts) other core-only code, quite outside of 0.72.

As such, patching D::S to fix this is a mere band-aid that would have to be replicated by every other module and program that uses $#array.

The obvious, simple and correct solution to that problem is to regress the core change that promiscuously converted every reference to $#somearray into a costly (and often pointless) application of 'magic', whether it was required or not.

Viz.

0] Perl> @a = map [], 1 .. 1e6;;
0] Perl> ## At this point, the process is using 195MB RAM.;;
0] Perl> $s = $#{ $_ } for @a;;
0] Perl> ## At this point, the process is using 395MB RAM.;;

It is patently f****** ridiculous that simply checking the size of an array, costs 200 bytes of memory.

Even more ridiculous, is that the raiser of the "bug", who is entirely competent to be aware of this, couldn't see past his own prejudices, and raised the "bug" against D::S, rather than as a regression against the core. But that is just par for the course.

And if that sounds bitter, it's because it is. And that tempts me to post a meditation that has been brewing in my head for 3 or 4 years now. Provisional title: "This community is as fractured and dysfunctional--and living in the same denial--as the USSR was in the late 1989 and early 1990."

If you want to read that meditation, then it is not enough to vote yah or nah. You will also have to encourage all your friends to vote. Because that post will only materialise if the absolute(*) vote of this post is >50.

Ie. Unless the vote tally for this post reaches greater than +|- 50, I will not make the meditation.

(*)Explanation of the logic behind making this an absolute limit available by /msg on request.