Are you sure that the database is as optimized as it can
be? Based on what is pointed out
I would suggest trying to make it a BTREE instead of a hash.
in reply to Performance quandary
I also note that your performance figures strike me as very
odd. As you say, your figures shouldn't scale that
badly, and dbms haven't when I have tried it. Based on
that fact I would look for things you might be doing that
would result in very large values that you keep on fetching
back and writing (so that the dbm may be finding them very
efficiently, but you would be writing them very slowly).
And what I see is that you are keeping lots of data in the
values, and that data set is constantly growing as you add
Right there I see evidence of a bad algorithm. Rather than
storing key/value pairs with large amounts of information
in the values which you keep on fetching and manipulating,
you would like to have a more complex data structure which
you can add to with less work. Do this and you should have
no scalability problems at all. Conversely if you try to
write in C and continue to use this data structure, you will
hit the same performance problem.
As for how you want to implement your data structure, that
is up to you. I would consider this to be a good "proof of
concept" project for DBI with DBD::SQLite.
Alternately you can sit down with a good data structures
and algorithms book and try to roll your own data structure
that scales better to large numbers of kids. For instance
you might want to store in the parent an entry with some
structured information, one of the pieces of which is how
many kids you have. To add a kid, pull this out, parse it,
increment the number of kids, add an entry with a key like:
"$parent_mdb|$kid_no" in which you store the kid, and store
the parent again. Sure, you have to edit two entries, but
both are small so you get the performance you had when you
had few entries and don't ever degrade.