Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Finding memory usage of perl structure efficiently

by dpavlin (Friar)
on Jun 26, 2010 at 15:32 UTC ( #846678=perlmeditation: print w/ replies, xml ) Need Help??

I always wanted to know memory usage of particular structure in my perl programs. Recently, I noticed that Storable file size corresponds nicely to memory usage of my application, so I used that as a hint of memory usage before loading data.

Then, I decided to show usage of already loaded data. Using Devel::Size total_size seemed like the way to go, but it's much slower than simple Storable piping to /dev/null

Rate total_size storable total_size 4.04/s -- -70% storable 13.5/s 234% --
using this code
use Benchmark qw(:all); use Devel::Size; use Storable; sub _storable_size { open(my $fh, '|-', 'cat > /dev/null'); Storable::store_fd $_[0], $fh; tell($fh); } my $o; $o->{$_} = rand() foreach ( 'a' .. 'z' ); my $hash; $hash->{$_} = $o foreach ( 0 .. 100_000 ); cmpthese( 100, { 'total_size' => sub { Devel::Size::total_size( $hash ) }, 'storable' => sub { _storable_size( $hash ) }, });
To be honest, I didn't expect so much difference. I can probably setup pipes myself to get a bit more performance, but I would really like to know alternative ways to find out perl memory usage without performance impact.

I'm experimenting with this code in gist on github as a log of things that I tried.


2share!2flame...

Comment on Finding memory usage of perl structure efficiently
Select or Download Code
Re: Finding memory usage of perl structure efficiently
by zwon (Monsignor) on Jun 26, 2010 at 19:07 UTC

    You can rewrite _storable_size as follows:

    sub _storable_size { length Storable::freeze $_[0]; }
      This is simplier, but it would take additional memory to freeze my structure (which is ~120Mb or so)

      2share!2flame...
Re: Finding memory usage of perl structure efficiently
by BrowserUk (Pope) on Jun 26, 2010 at 19:44 UTC

    Just a thought. Have you tried Devel::Size v 0.72?

    You should find it uses far less memory for it own purposes, which means it runs much more quickly. It also gives more accurate results and is far less likely to cause an exception than the previous versions.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      It's not easy to find as it's marked as "unauthorized". search.cpan.org or cpan shell search does not show it. You need to specifically search for it to find. So, why not re-release this as an official 0.73 as Tels already marked this distro as abandonware{1}? Apart from improvements, I know that you also fixed some serious bugs in it.
        It's not easy to find as it's marked as "unauthorized". search.cpan.org or cpan shell search does not show it.

        When I look at Devel::Size, 0.72 is linked as the "latest release".

        So, why not re-release this as an official 0.73 as Tels already marked this distro as abandonware{1}?

        I'm not sure how that would be any different from having already released it as version 0.72?

        I did offer to take it over, but apparently the existing two co-maintainers couldn't agree to that.

        Apart from improvements, I know that you also fixed some serious bugs in it.

        At the point where I released what I had after failing to get it adopted as "official" (whatever that means), I believe I had fixed every outstanding bug on RT, including several that had been around for years; I added exception handling to avoid the frequent traps; and made it 64-bit compilable.

        Dunno what more I could have done?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks for lot for a pointer to newer version. On first run, it reported
      Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814
      Fiar enough, but overall it's fastest method to find size of perl structure:
      # 20000 elements Rate size_0.71 reopen cat size_0.72 freeze size_0.71 80.3/s -- -43% -65% -83% -84% reopen 142/s 77% -- -38% -71% -71% cat 227/s 183% 60% -- -53% -54% size_0.72 485/s 505% 242% 114% -- -1% freeze 490/s 511% 245% 116% 1% -- # 30000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 37.1/s -- -54% -65% -83% -84% reopen 80.4/s 117% -- -25% -63% -66% cat 107/s 189% 33% -- -51% -54% freeze 217/s 486% 170% 103% -- -7% size_0.72 234/s 530% 191% 118% 7% -- # 40000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 22.1/s -- -59% -67% -83% -85% reopen 54.1/s 144% -- -20% -58% -64% cat 67.4/s 204% 25% -- -48% -55% freeze 130/s 487% 141% 93% -- -13% size_0.72 149/s 571% 175% 121% 14% -- # 50000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 13.6/s -- -63% -70% -81% -85% reopen 36.5/s 169% -- -20% -48% -61% cat 45.8/s 237% 25% -- -35% -51% freeze 70.2/s 417% 92% 53% -- -24% size_0.72 92.6/s 582% 153% 102% 32% -- # 60000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 9.97/s -- -66% -69% -80% -86% reopen 28.9/s 190% -- -10% -43% -59% cat 31.9/s 220% 11% -- -37% -55% freeze 50.5/s 407% 75% 58% -- -28% size_0.72 70.4/s 606% 144% 120% 39% -- # 70000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 7.91/s -- -65% -69% -79% -86% reopen 22.7/s 187% -- -12% -41% -59% cat 25.9/s 227% 14% -- -32% -54% freeze 38.2/s 383% 68% 48% -- -32% size_0.72 56.1/s 609% 147% 117% 47% -- # 80000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 5.78/s -- -69% -72% -80% -86% reopen 18.9/s 227% -- -10% -35% -55% cat 21.0/s 264% 11% -- -28% -50% freeze 29.1/s 404% 54% 39% -- -31% size_0.72 42.3/s 632% 124% 101% 45% -- # 90000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 4.95/s -- -69% -72% -80% -86% reopen 16.1/s 226% -- -9% -35% -55% cat 17.7/s 257% 9% -- -29% -51% freeze 24.7/s 399% 53% 40% -- -32% size_0.72 36.1/s 630% 124% 105% 46% -- # 100000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 4.21/s -- -69% -72% -80% -86% reopen 13.8/s 227% -- -9% -35% -56% cat 15.2/s 261% 10% -- -29% -51% freeze 21.3/s 407% 55% 40% -- -31% size_0.72 31.0/s 637% 125% 104% 45% --
      I would love to see version 0.71 depreciated :-)

      2share!2flame...

        On first run, it reported Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814

        I assume that you are running a 64-bit platform with 8GB of ram?

        My version requires a small, static area of memory for tracking purposes. This has to be sized to cater for the size of the memory of the machine. It ships with this sized to cater for the maximum memory on a 32-bit machine (8192 * sizeof(void *).

        On a 64-bit machine that has more than 4GB, it requires this be increased.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://846678]
Approved by toolic
Front-paged by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (6)
As of 2014-07-24 03:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (157 votes), past polls