Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Compact data classes

by BrowserUk (Pope)
on Jun 08, 2013 at 06:07 UTC ( #1037797=note: print w/ replies, xml ) Need Help??


in reply to Compact data classes

This uses 4.4MB of ram to construct 15,000 objects each of 10 x 15-char fields.

The array holding those 15k objects (containing a total of 2.15MB of string data) occupies 3.67MB of ram.

The accessors are lvalue subs and their performance should be comparable with any other perl OO objects.

No validation is done.

use Class::Struct::Compact; use Devel::Size qw[ total_size ]; use Data::Dump qw[ pp ]; my @chars = ( 'a'..'z' ); sub dummy { my $n = shift; join '', @chars[ map int( rand @chars ), 1 .. $n ]; } Class::Struct::Compact->new( 'Test', F1 => 15, F2 => 15, F3 => 15, F4 => 15, F5 => 15, F6 => 15, F7 => 15, F8 => 15, F9 => 15, F10 => 15, ); printf "Class constructed: check mem: "; <>; our $N //= 15_000; my @db; push @db, Test->new( dummy( 150 ) ) for 1 .. $N; printf "Instances created: check mem: "; <>; @db = sort{ $a->F1 cmp $b->F1 } @db; #pp \@db; printf "Instances sorted: check mem: "; <>; print "total size: ", total_size \@db; for( 0, $#db ) { print "Record $_"; print "\t", $db[ $_ ]->F1; print "\t", $db[ $_ ]->F2; print "\t", $db[ $_ ]->F3; print "\t", $db[ $_ ]->F4; print "\t", $db[ $_ ]->F5; print "\t", $db[ $_ ]->F6; print "\t", $db[ $_ ]->F7; print "\t", $db[ $_ ]->F8; print "\t", $db[ $_ ]->F9; print "\t", $db[ $_ ]->F10; } __END__ C:\test>perl \perl64\site\lib\Class\Struct\Compact.pm Class constructed: check mem: 4.6MB Instances created: check mem: 9.0MB Instances sorted: check mem: 9.1MB total size: 3851232 Record 0 aaartlhgpkapaol jrbkelwlfklkjgn rhdrrltzezyuenc zxccfpxpbzcxoqy ysfqlfkrnhmaqhf vclpccofujbyars gwrdngknxjyxxni foiuaojwzrqouzc msbepsdptomdtbe qazhgrkywspzsts Record 14999 zzypdjmcgmgxnso yzygmkgabelvqlj xihybqagfiydipo fpgsaybyhrfawuc zyxekczeaxfomrs lwyannakxmgists nzehvwysfvpkeuf gggdblbshwhgnto vtdgbjvgwevpurx wtdgjumncxgfaih

The module & test code:


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re: Compact data classes
Select or Download Code
Re^2: Compact data classes
by creeble (Sexton) on Jun 08, 2013 at 21:55 UTC
    omg, you may be my best friend. Let me check that and get back.
Re^2: Compact data classes
by creeble (Sexton) on Jun 09, 2013 at 17:04 UTC
    Wait, where is this magical Class::Struct::Compact? It's not in CPAN afaict?

      As hdb points out, it's in the spoiler at the end of my post.

      It's not on cpan because I wrote it -- actually adapted it from some existing code -- in reponse to your OP.

      I'd want to use a few times myself and see what else it needs before putting it out for general use. For starters it needs some error checking and Carp for when things go wrong.

      It'd also be nice to use pack templates to allow for numeric fields; but then it I'd have to drop the lvalue-ness of the accessors.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Doh! I see it. This is very clever; compiled class and substr(). The only apparent downside is the fixed record length, but I could probably find a workable value for them. Given the roughly 5x overhead of Class::Struct, the savings ought to be well worth it. Let me PM you about error checking, etc. Thanks so much.
Re^2: Compact data classes
by creeble (Sexton) on Jun 10, 2013 at 17:06 UTC
    (I just love it when I reply thinking I'm logged in, ugh.)

    This is very clever! The only drawback is the fixed record length, but the savings is so significant compared to Class::Struct's ~5x overhead I should be able to find a workable size.

    Let me PM you about error checking, etc. Thanks so very much.

Re^2: Compact data classes
by creeble (Sexton) on Jun 10, 2013 at 17:56 UTC
    Okay, one more question for BrowserUK here which may be of benefit to other similarly-clueless sub-monks like myself.

    Why does your solution use a compiled (eval) class? Is this faster or does it save memory vs. a solution like roboticus proposes?

      Because it allows me to 'hardcode' the numbers in the substrs.

      If you uncomment the print before the eval, you'll see something like:

      package Test; use constant { F1_N => 0, F2_N => 15, F3_N => 30, F4_N => 45, F5_N => +60, F6_N => 75, F7_N => 90, F8_N => 105, F9_N => 120, F10_N => 135, } +; use constant { F1_L => 15, F2_L => 15, F3_L => 15, F4_L => 15, F5_L => + 15, F6_L => 15, F7_L => 15, F8_L => 15, F9_L => 15, F10_L => 15, }; sub new { my $class = shift; my $self = shift // ''; return bless \$self, $class } # line 1 "sub_F4" sub Test::F4 :lvalue { my $self = shift; substr( $$self, F4_N(), F4_L() ); }

      F4_N() and F4_L() are constant subs which get optimised away during compilation, leaving hardcoded numbers which are faster than variables.

      The memory saving comes from packing the fields into single strings; the performance comes from asking the sub to do as little as possible.

      That said, by explaining that, I've spotted another couple of optimisations; and a potential bug. I'll get back to you with a revised version 2 days from now.

      (A good reason for not uploading to cpan straight away :)


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Okay, I totally understand that now; I had un-commented the print and not remembered that the 'use constant' actually does get optimised away at compile time. Very cool.

        FWIW, I don't need the lvalue-ness of the accessors, setting them with param(value) makes me totally happy, especially if it saves anything. I'm embarrassed to say I've never seen the ":lvalue" attribute in use, which is rather nifty.

        This is so promising. I really appreciate your expertise on this; I've tried various solutions like roboticus proposed, but performance has been unusable.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1037797]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (9)
As of 2014-12-20 04:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (95 votes), past polls