If your goal is to have people not judge quickly, shouldn't the same risk be entailed to *any* vote, whether it's a down-vote or an up-vote?
How about this: all votes you make are public. Anyone can challenge your vote in the 24 hours following your vote. Once your vote has been challenged, you got 24 hours to write down the motivation behind it. People get to vote on that. If that doesn't attract enough up votes more than it has down votes, you lose. The first time you lose, you lose all your XP (and become leve 1 again). The second time you lose, you lose your posting rights for a year. The third time, Vroom comes over, pulls out your nails, kills your dog, sets your house on fire, slashes the tires from your truck, and sells your kids into slavery.
That ought to stop voting. So no more down votes.