But passwords are typically rather short so not too difficult to crack by brute force.
What you are describing here are not passwords in general but poor passwords. Good passwords are typically rather long so too difficult to crack by brute force.
Assuming the password is made up only of upper case, lower case letters and numbers then there are only (!) 218,340,105,584,896 permutations. That is 628.
Again, good passwords do not just consist of letters and digits. Even if they did it seems you are assuming just Roman letters and Arabic digits. There are plenty of other character sets from which to choose.
If you don't want to get pwned, don't use poor passwords. If you don't want your users to get pwned don't let them use poor passwords.