in reply to Hash lookups, Database lookups, and Scalability

First, DBD::SQLite is not a sound database any way. Also there is a big performance boost when you wrap your queries in DBD::SQLite's transaction. I didn't see you use it, and it would be interesting to get another benchmark with that enabled.

In general, database search is less efficient than in-memory ones. First you have more freedom to tailor your algorithm and data structure for in-memory search; secondly for database search, the results are transferred through network communication, two-way communication for each query, which obviously costs extra time. How good is the database's cache ability will also impact the performance.

However this does not mean that database search is not a good viable solution. Database gives you lots of important benefits than just speed, which is OT here.