in reply to Performance quandary
I also note that your performance figures strike me as very odd. As you say, your figures shouldn't scale that badly, and dbms haven't when I have tried it. Based on that fact I would look for things you might be doing that would result in very large values that you keep on fetching back and writing (so that the dbm may be finding them very efficiently, but you would be writing them very slowly). And what I see is that you are keeping lots of data in the values, and that data set is constantly growing as you add kids.
Right there I see evidence of a bad algorithm. Rather than storing key/value pairs with large amounts of information in the values which you keep on fetching and manipulating, you would like to have a more complex data structure which you can add to with less work. Do this and you should have no scalability problems at all. Conversely if you try to write in C and continue to use this data structure, you will hit the same performance problem.
As for how you want to implement your data structure, that is up to you. I would consider this to be a good "proof of concept" project for DBI with DBD::SQLite.
Alternately you can sit down with a good data structures and algorithms book and try to roll your own data structure that scales better to large numbers of kids. For instance you might want to store in the parent an entry with some structured information, one of the pieces of which is how many kids you have. To add a kid, pull this out, parse it, increment the number of kids, add an entry with a key like: "$parent_mdb|$kid_no" in which you store the kid, and store the parent again. Sure, you have to edit two entries, but both are small so you get the performance you had when you had few entries and don't ever degrade.
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Re (tilly) 1: Performance quandary
by SwellJoe (Scribe) on Feb 24, 2002 at 04:12 UTC |