|Pathologically Eclectic Rubbish Lister|
It may very well be that whatever you come up with is only faster than File::Find on some setups of disks/volume managers/filesystem, and slower on others.
You need to benchmark to find out.
Now, in theory, carefully handcrafting something that does exactly what you need is going to be faster than a more general setup than File::Find. But whether that's actually be measurable is a different question.
Of course, as you describe the problem, the bottleneck might very well be your I/O.
Have I said you should benchmark? No? Well, benchmark!