in reply to Optimising processing for large data files.
Databases are never quicker unless you can use some fairly simplistic criteria to make wholesale reductions in the volume of the data that you need to process within your application program.
I recommend using databases to people not because of any kind of performance gain you get from the "database magic bullet", but because there's a lot of programming work you can cut out of the picture.
For instance, if you need a data structure that can persist outside of your program, and be accessed and modified via multiple programs, all you have to do is create a table to represent the data structure, and all of the work is basically done for you. SQL is all you need, and it's done very easily for someone who is familiar with it.
However, if you're rolling your own, it's going to take a lot of time, you're going to have to take care of a lot of details which are just provided for you using a database, and (if you're on a virtual host without root access) it may be the only thing you have write access to.
Want to support the EFF and FSF by buying cool stuff? Click here