I like to think of it this way:

*It's my vote, and I'll downvote you if I want to.*

So yes, if there's anything I don't like about the answer, I might vote it down, whether it is incorrect or merely displeasing to my personal sense of aesthetics.