You have my sympathies, but I think you're going after the wrong problem. I've been on the receiving end of that treatment myself -- I was daring to argue with someone who was clearly initially down-voting everything I said, in addition to responding to me -- but the thing that was most interesting about that was the pattern of up-votes happening with the person I was arguing with. Everything, even trivia, seemed to be getting a few up-votes: I strongly suspect he's using sock-puppets to vote for his own postings.
The fact that we don't have any sort of real, verified I.D.s means that the entire XP system is essentially a toy, which really can't be taken seriously. You can hack on it all you want-- eliminate downvotes, forbid voting in threads you're participating in, add "meta-moderation", whatever-- but it'll remain an easily gameable toy.