If you don't want to reward the user for answering a low-quality question it's best to leave the answer alone, not downvote it, especially if you know that it is a good, correct answer. However, if a question is so unclear that you can't even judge if the answer is correct, it's also best to leave it alone.
Personally I only upvote an answer if I can determine that it is useful and correct, and downvote it if I can determine that it is bad, wrong or completely misses the point of the question (if I can understand the question in the first place). Otherwise I leave it alone for either of the above reasons.
As a little tidbit, I recently witnessed a case of an answer getting downvoted, except the downvoter was unable to adequately explain the vote because the question was so unclear. The user posting the answer got pretty annoyed that we voted to close the question, but absolutely nobody bothered to edit the question to try and make it clearer — not even the asker, after being prompted from multiple people, including the user posting the answer — and not even the answerer either, who apparently understood it enough that he should have been able to help with an edit.
Turns out the answer was indeed correct, as it was then accepted. Weird, huh?
And here's a tip for answerers: if you understand a question and its answer well enough, please do the community a favor and:
- Ask the asker to edit the question to clarify it
- Failing that, try to clarify it as much as you feel you know how, especially if the community makes it clear that it's in a dire need of an edit.
An edit can go a long way — in many cases it helps immensely in getting a question reopened, and sometimes it may even boost the question's score, if there are enough viewers.
Also a tip for everyone: when in doubt, don't. We want good answers to good questions, not unconfident answers to unclear questions. Guesswork isn't always going to work out for the asker, the answerer(s) as well as (and especially) future readers. This is why we have the NARQ close reason at all, and even when a question does get closed, it can always be reopened after it's improved through an appropriate edit.