Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Finding memory usage of perl structure efficiently

by dpavlin (Friar)
on Jun 27, 2010 at 13:20 UTC ( #846776=note: print w/ replies, xml ) Need Help??


in reply to Re: Finding memory usage of perl structure efficiently
in thread Finding memory usage of perl structure efficiently

Thanks for lot for a pointer to newer version. On first run, it reported

Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814
Fiar enough, but overall it's fastest method to find size of perl structure:
# 20000 elements Rate size_0.71 reopen cat size_0.72 freeze size_0.71 80.3/s -- -43% -65% -83% -84% reopen 142/s 77% -- -38% -71% -71% cat 227/s 183% 60% -- -53% -54% size_0.72 485/s 505% 242% 114% -- -1% freeze 490/s 511% 245% 116% 1% -- # 30000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 37.1/s -- -54% -65% -83% -84% reopen 80.4/s 117% -- -25% -63% -66% cat 107/s 189% 33% -- -51% -54% freeze 217/s 486% 170% 103% -- -7% size_0.72 234/s 530% 191% 118% 7% -- # 40000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 22.1/s -- -59% -67% -83% -85% reopen 54.1/s 144% -- -20% -58% -64% cat 67.4/s 204% 25% -- -48% -55% freeze 130/s 487% 141% 93% -- -13% size_0.72 149/s 571% 175% 121% 14% -- # 50000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 13.6/s -- -63% -70% -81% -85% reopen 36.5/s 169% -- -20% -48% -61% cat 45.8/s 237% 25% -- -35% -51% freeze 70.2/s 417% 92% 53% -- -24% size_0.72 92.6/s 582% 153% 102% 32% -- # 60000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 9.97/s -- -66% -69% -80% -86% reopen 28.9/s 190% -- -10% -43% -59% cat 31.9/s 220% 11% -- -37% -55% freeze 50.5/s 407% 75% 58% -- -28% size_0.72 70.4/s 606% 144% 120% 39% -- # 70000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 7.91/s -- -65% -69% -79% -86% reopen 22.7/s 187% -- -12% -41% -59% cat 25.9/s 227% 14% -- -32% -54% freeze 38.2/s 383% 68% 48% -- -32% size_0.72 56.1/s 609% 147% 117% 47% -- # 80000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 5.78/s -- -69% -72% -80% -86% reopen 18.9/s 227% -- -10% -35% -55% cat 21.0/s 264% 11% -- -28% -50% freeze 29.1/s 404% 54% 39% -- -31% size_0.72 42.3/s 632% 124% 101% 45% -- # 90000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 4.95/s -- -69% -72% -80% -86% reopen 16.1/s 226% -- -9% -35% -55% cat 17.7/s 257% 9% -- -29% -51% freeze 24.7/s 399% 53% 40% -- -32% size_0.72 36.1/s 630% 124% 105% 46% -- # 100000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 4.21/s -- -69% -72% -80% -86% reopen 13.8/s 227% -- -9% -35% -56% cat 15.2/s 261% 10% -- -29% -51% freeze 21.3/s 407% 55% 40% -- -31% size_0.72 31.0/s 637% 125% 104% 45% --
I would love to see version 0.71 depreciated :-)

2share!2flame...


Comment on Re^2: Finding memory usage of perl structure efficiently
Select or Download Code
Re^3: Finding memory usage of perl structure efficiently
by BrowserUk (Pope) on Jun 27, 2010 at 15:30 UTC

    On first run, it reported Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814

    I assume that you are running a 64-bit platform with 8GB of ram?

    My version requires a small, static area of memory for tracking purposes. This has to be sized to cater for the size of the memory of the machine. It ships with this sized to cater for the maximum memory on a 32-bit machine (8192 * sizeof(void *).

    On a 64-bit machine that has more than 4GB, it requires this be increased.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      I'm using 64-bit kernel with 32-bit userland. It might make sense to bump it up to 16384 so 64-bit users won't have to touch source at expense of small memory increase.

      2share!2flame...

        A year and a half ago when I release 0.72, the preponderance of people were running 32-bit and 8192 was already double what they required (most 32-bit processes are limited to 2GB and 8192 allows for 4GB). And even most 64-bit systems maxed at 4GB.

        But, Moore's law moves inexorably on, and 8GB is probably about right to cover the next couple of years or so. If I decide to release a forked version, I will increase the default to 16384. I have been waiting for the appearance of *a* bug against the current version, but none has emerged..

        There is a bug report, raised against 0.71, that also affects 0.72, but I consider that a bug (or rather, a regression), in the perl core. This because the the "bug" did not happen prior to the 5.10 release; and because it affects (and afflicts) other core-only code, quite outside of 0.72.

        As such, patching D::S to fix this is a mere band-aid that would have to be replicated by every other module and program that uses $#array.

        The obvious, simple and correct solution to that problem is to regress the core change that promiscuously converted every reference to $#somearray into a costly (and often pointless) application of 'magic', whether it was required or not.

        Viz.

        0] Perl> @a = map [], 1 .. 1e6;; 0] Perl> ## At this point, the process is using 195MB RAM.;; 0] Perl> $s = $#{ $_ } for @a;; 0] Perl> ## At this point, the process is using 395MB RAM.;;

        It is patently f****** ridiculous that simply checking the size of an array, costs 200 bytes of memory.

        Even more ridiculous, is that the raiser of the "bug", who is entirely competent to be aware of this, couldn't see past his own prejudices, and raised the "bug" against D::S, rather than as a regression against the core. But that is just par for the course.

        And if that sounds bitter, it's because it is. And that tempts me to post a meditation that has been brewing in my head for 3 or 4 years now. Provisional title: "This community is as fractured and dysfunctional--and living in the same denial--as the USSR was in the late 1989 and early 1990."

        If you want to read that meditation, then it is not enough to vote yah or nah. You will also have to encourage all your friends to vote. Because that post will only materialise if the absolute(*) vote of this post is >50.

        Ie. Unless the vote tally for this post reaches greater than +|- 50, I will not make the meditation.

        (*)Explanation of the logic behind making this an absolute limit available by /msg on request.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://846776]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (14)
As of 2014-08-20 20:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (124 votes), past polls