|Perl: the Markov chain saw|
I am speccing out a change to E2, but wanted to run it past some perl-heads so I figured I'd toss it out here for a few ideas. I'm looking for a way to buy myself some more database cycles with which to expand, and doing some nodecache analysis, I think that we can tailor both of our sites by changing the way that the nodecache works to perform delta-encoded update checking, rather than performing an albeit cheap but uberfrequent query on the database
First, we'd create a table, say called delta_nodeversion. This would look like a HEAP table with timestamp,node_id as a key. Everytime we call Everything::NodeCache::incrementGlobalVersion and the type exists in a pre-defined perma-cache list, we would drop the node_id into delta_nodeversion.
Next, every time the page is run (inside of resetCache), it will check the delta_nodeversion for any changes since its last check time it has run. If it has found them, it will immediately go and tombstone those queue items, and let the page run normally. The Everything::NodeCache::isSameVersion then returns 1 for anything in the permanent queue map. When there's nothing in the queue, a big update day would only harm performance a bit, or one could restart your webheads in a controlled fashion and trim the queue afterwards if you were worried about it.
For performance, one might prune the table for entries older than a day, since it's unlikely that all of the apache threads have not all run once in a day.
htmlcodes,htmlpages,superdocs,maintenances,settings,containers,nodetypes,themes and themesettings might all be good candidates for this type of treatment, depending on how much load you want to push from your database onto your memory stack. The basis of this performance increase is based on the theory that it is cheaper to check one table that you can keep small, with a slightly more expensive query, than to check another table 15 times per page.I'd welcome the feedback on it, thanks