Yes, people do occasionally -- (downvote) a node. Their reasons are as varied as their haircuts, and make about as much sense. The best thing to do is to not get hung up on your nodes' reputation. Be more concerned about your personal reputation!
Changes in your nodes' reputation can affect your XP... but again, this barometer of personal worth should not be given too much weight; it's really just a game we play to make life at the Monastery more interesting.
As the nice people at Everything have noted:
Historically, posts complaining about or suggesting radical changes to the Voting/Experience System are themselves unpopular; most get downvoted and many get reaped.
The administration does not take the voting and experience point system all that seriously.
Woe unto those who do.
XP is an imaginary number granted to you by an anonymous stranger.
tye has written a nice FAQ-like node which could be (but isn't) entitled Why does it seem like I have an enemy who is systematically downvoting every node I post?