You'd be right if people randomly picked a password from the entire key space (picking with a uniform distribution). But if you allow passwords with a length of 4 - 10000 characters, with no restriction on the character set, there will be people that pick a four character password, with all lowercase ASCII letters.
And attackers know that password picking cracking is more than math. Psychology plays a role as well. They will try "dictionary" attacks first, because people pick short existing words far more often than you would get by picking a random element from the total set of allowed passwords.