No. You should vote on answer regardless of the quality of the question (e.g. downvote if it is incorrect). If the question is off-topic, close/flag the question.
This thread is now outdated, please see https://meta.stackoverflow.com/a/255460/ for up-to-date policy guidelines.
But since some people are still voting on this answer many years later, let me explain the reasoning behind this a bit. In my view the most valuable resource in the possession of Stack Overflow, is the large number of volunteers that answer the new questions as they are asked. This process is what ensures that the content on the site remains up-to-date with the current best practices.
Some people complain that by answering "bad" questions, they encourage more of them to be asked. But even so, I believe that we still shouldn't try to discourage well-intentioned answerers. The cost of losing potentially valuable contributors is too great. Instead of downvoting, we can refrain from upvoting, and maybe add a comment if the question is completely out of the scope of the site.
If someone keeps answering "bad" questions anyway, and it bugs you too much, vote to delete the question if it gets closed, and if deleted, the answer will go down the drain too, so he will lose any points he gained by it.