Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
Are you sure that the database is as optimized as it can be? Based on what is pointed out here I would suggest trying to make it a BTREE instead of a hash.

I also note that your performance figures strike me as very odd. As you say, your figures shouldn't scale that badly, and dbms haven't when I have tried it. Based on that fact I would look for things you might be doing that would result in very large values that you keep on fetching back and writing (so that the dbm may be finding them very efficiently, but you would be writing them very slowly). And what I see is that you are keeping lots of data in the values, and that data set is constantly growing as you add kids.

Right there I see evidence of a bad algorithm. Rather than storing key/value pairs with large amounts of information in the values which you keep on fetching and manipulating, you would like to have a more complex data structure which you can add to with less work. Do this and you should have no scalability problems at all. Conversely if you try to write in C and continue to use this data structure, you will hit the same performance problem.

As for how you want to implement your data structure, that is up to you. I would consider this to be a good "proof of concept" project for DBI with DBD::SQLite.

Alternately you can sit down with a good data structures and algorithms book and try to roll your own data structure that scales better to large numbers of kids. For instance you might want to store in the parent an entry with some structured information, one of the pieces of which is how many kids you have. To add a kid, pull this out, parse it, increment the number of kids, add an entry with a key like: "$parent_mdb|$kid_no" in which you store the kid, and store the parent again. Sure, you have to edit two entries, but both are small so you get the performance you had when you had few entries and don't ever degrade.

In reply to Re (tilly) 1: Performance quandary by tilly
in thread Performance quandary by SwellJoe

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others chanting in the Monastery: (5)
    As of 2018-01-20 17:35 GMT
    Find Nodes?
      Voting Booth?
      How did you see in the new year?

      Results (227 votes). Check out past polls.