I can't help thinking you should ask Google how to perform this sort of task. Rapid lookup of mega-large databases is their bread and butter.
Speaking of databases, that's probably a part of the solution. Distil the data into a suitably indexed database, then use that to perform the searches. Maybe if you tell us something of the why we can give somewhat more focused advice. If you've actually tried to solve the problem yourself you might like to tell us what you've tried and where it came unstuck.
True laziness is hard work