Yes and yes. No. That is...
We have to track every single vote otherwise there would be no way for the system to work the way it does (preventing you from voting on the same node twice, etc.). Each vote record must record who voted and what node was voted on. They also record how you voted (++ or --) and when. After a long run of amazing abuses of the voting system, we added the node author (the "votee") to these records and added a couple of indices. Now it is somewhat fast and easy to review how someone has been voting or what votes have recently been received by someone (or to summarize information such as what percentage of recent votes were downvotes or what percentage of recent downvotes were cast by the top N down-voters, etc.). We had to do this to get the abuses under control. I still do this occasionally when I see complaints, in part to try to figure out how to improve things.
Each node has a 'reputation' that is the sum of ++ votes minus the sum of -- votes and a 'votescast' that is the sum of all votes. From these two values it is easy to compute the number of just ++ votes and the number of just -- votes.
rep = up - down; cast = up + down
up = rep + down; down = cast - up
up = rep + cast - up; down = cast - rep - down
2*up = rep + cast; 2*down = cast - rep
up = ( rep + cast ) / 2; down = ( cast - rep ) / 2