Dunno if it already works this way, but one option might
be to have a downvote 'cost' something; either it uses up
one of your votes but doesn't enter the XP lottery or
perhaps the reverse; a -- puts you in the negative XP
lottery (really bad odds?) so that it might take one
of your own XP also. In the end, who cares, you'll be
a perl monk regardless of your XP but on that other hand,
there's the node now in code which isn't great but does it
deserve a -5 just because its not wonderful?
Update: cowards ...