If your search radius is so small (25 miles), You could eliminate 90% of your data before you do a lot of the number crunching using a less-accurate estimate first -- perhaps a good choice would be to check if the latitude/longitude are within ~10 degrees (or some constant appropriate for your data) of the origin and skip the coordinates if they are not. Maybe just comparing the number of degrees is a bad choice, but I also know that there are several great circle distance calculation algorithms around, with varying trade-offs between speed and accuracy. The key is to eliminate bad matches first with the low-accuracy algorithm, then use the high-accuracy one on the close ones.
You may also wish to check out Geo::Distance or Geo::PostalCode to see how they do it. Geo::PostalCode claims to have a feature exactly as you describe.