I don't really mind either way.
However, I don't think that "get rid of the spend-all-your-votes bonus" is going to do much to correct this problem (if problem it is). I think that most of the flippant voters do it because they might get one XP just for voting, not because they'll get XP later. (I will admit to having gone to quite some lengths to spend leftover votes for just this reason, although I like to think that I usually try to find suitably deserving nodes. There are plenty of those around.) Now we're back to rozallin's proposal, which may not be a bad thing.
Yours in pedantry,
"Anything you put in comments is not tested and easily goes out of date." -- tye