I think this is because in my head I'm unconsciously thinking of the number of votes as a 'rating' or 'score' indicating the quality of the answer
I do this consciously! What matters most is that the best answer is the first answer, or at least fairly near the question. If it's not, I'll vote in a way that pushes it in that direction. So yeah, I generally don't waste votes on decent, but already highly-ranked answers, and I'll often down-vote other answers to a question if it'll help the answer I consider "best" float to the top.
I make an exception for exceedingly subjective, non-CW Q&A though, since there's no real point in trying to "convert" others to my opinion by voting. I'll occasionally down-vote all non-CW answers though, as sort of a "heads up: are you sure you wanted to answer this"?
I've been thinking more carefully about what the vote represents
My votes represents my opinions as to which answers I like best. That is all.
