I think this is because in my head I'm unconsciously thinking of the number of votes as a 'rating' or 'score' indicating the quality of the answer
I do this consciously! What matters most is that the best answer is the first answer, or at least fairly near the question. If it's not, i'llI'll vote in a way that pushes it in that direction. So yeah, iI generally don't waste votes on decent, but already highly-ranked answers, and i'llI'll often down-vote other answers to a question if it'll help the answer iI consider "best" float to the top.
I make an exception for exceedingly subjective, non-CW Q&A though, since there's no real point in trying to "convert" others to my opinion by voting. I'll occasionally down-vote all non-CW answers though, as sort of a "heads up: are you sure you wanted to answer this"?
I've been thinking more carefully about what the vote represents
My votes represents my opinions as to which answers iI like best. That is all.