To be able to do a "trusty vote" does it really matter how much XPs you have? Bottom line, I say keep the voting XPs but don't
increase the number of votes as you "grow".
Or, increase it, but not by as much as it is now. After all,
the idea of giving you more votes as you grow is the assumption
that as you have more experience, you are more qualified
to "grade" other people's comments, which makes sense.
I just made monk, and 16 votes per day is plenty to spend.
Maybe the increase could be by 2-3 votes per rank, instead of
by 3-5 as it is now.