I have done this in the past with Berkeley DB and had much success.
in reply to using indexed (dbm) files
I had a CGI written in C (way back in the day) that would write to the Berkeley DB. Then my perl script would come along every so often and yank the data out of the Berkeley DB for processing.
I don't have a lot of experience with the other options you listed, but Berkeley DB is a very robust system that will probably handle just about anything you need it for. And the folks at Sleepycat are great, too. :)