I like to think of it this way:
It's my vote, and I'll downvote you if I want to.
So yes, if there's anything I don't like about the answer, I might vote it down, whether it is incorrect or merely displeasing to my personal sense of aesthetics.
After all, that's the very nature of the sites: I've accumulated a certain amount of reputation by demonstrating responsibility, and in so doing have been conferred the ability to downvote. The site is trusting me to make these decisions, so I shall do so. After all, that's the very nature of the sites: I've accumulated a certain amount of reputation by demonstrating responsibility, and in so doing have been conferred the ability to downvote. The site is trusting me to make these decisions, so I shall do so.
More waxing poetical: I don't actually do this very often; more usually I will just leave a comment "tsk, you shouldn't do it this way!" and leave it at that, because I don't like those little red marks in my rep history. Or, when faced with a series of "correct" answers, I will only upvote the answer that both answered the questions and doesn't do anything evil. My point is that we all have the ability to downvote and we should feel free to use that ability with discretion.