This is based on an idea I had in chatter earlier.
I think the following might be useful. Once per day (week, some period), let people (perhaps only people of sufficient level) select one node of their own to challenge votes on. When the node is challenged all people who voted on it (either way) in the last week are notified. They have a week to send a /msg explaining their vote. After a week all votes that were not explained get removed from that node. Any person who meets some criteria for having too many votes rescinded (eg 5 in a row) loses their voting privileges.
This is an attempt at a compromise. It largely maintains the privacy of the current system, while providing a mechanism for people to get feedback about why they are getting voted on as they are. However the volume of feedback will be small, and people can always choose to maintain privacy about how they voted. But you cannot choose to vote both unaccountably and controversially.