For any given person, chances are that most of your posts have relatively low ratings. I know that's true of my posts, even for a lot of posts where I posted useful solutions, since apparently being first is far more important than being correct and useful when it comes to points. I can live with that, and even the occasional downvote, but statistically, it's pretty easy for one or two people with an axe to grind to take you down a large percentage of your rating. Systematic attacks need to either be stopped by the mods, or you should be able to see who's doing it so you can retaliate. Preferably the former, but the latter is acceptable as an alternative.
EDIT: Ok, I see that there's an update post here:
Don't know why the primary thread wasn't updated instead, but oh well. Apparently since I was here last, the system has been updated so you can't downvote ancient posts, and a single downvote won't count. So that at least somewhat reduces the problem with downvotes.