By monkeying around w/ your own searches, you're probably reinventing some wheels, and doing it rather ineffectively.

Any high access / high volume data should be indexed as a rule of thumb. Even moving your data into something like SQLite would give you the benefit of indexing.

If it were me, I'd definitely persue some sort of robust indexing mechanism first. Probably think about threading & smart serialization next.

Good luck -- that actually sounds like a fun piece of work.