|No such thing as a small change|
Caching DBM hash tieby Tardis (Pilgrim)
|on Jul 16, 2003 at 00:41 UTC||Need Help??|
Tardis has asked for the
wisdom of the Perl Monks concerning the following question:
I have a problem. In a nutshell I have a homegrown database application that uses a DBM tie to provide data access. Concurrency is handled by flock, only providing access to a single writer, or many readers at any time.
As the number of people using this application grows, it's not scaling too well. For obvious reasons. We need row level locking, or SQL or something.
To give you an idea of the seriousness, some write operations are taking upward of 30 seconds. That's 30 seconds where no one can use the entire system. That's bad for an interactive app, in anyones language :-)
Yes SQL is on the cards, but it will require a major rewrite, which we aren't really ready for yet. Half-assed SQL conversions (basically a DBM file in SQL) are also an option, but require some API changes.
I thought this morning of a change which would require a very small amount of changes to the API, with plenty of potential gains. It might also be generally useful for the perl community at large.
My idea is to provide a wrapper around a DBM tie that would provide the following extra functionality:
The delayed writes is a huge win though. If write operations can be deferred until the system is not busy, or until a certain amount of time has elapsed, interactive performance will improve markedly.
I imagine that the simplest implementation would be a module which talks to a named pipe or socket, which has a seperate script running on the other side. A nice failsafe could be that the client side of the thing could fall back to normal tie behaviour if the server side can't be contacted.
Firstly, am I insane to think this could be done well in pure perl?
Secondly, has anyone already done this? I can't see anything on CPAN, but maybe I'm not looking in the right places.
Lastly, I really, really need to be able to preserve my current behaviour where database operations are simply reads or writes to hash keys. I'm moving away from that but it's a slow process. Any solutions which change the way I'd read and write my data significantly are really useless to me at this stage.