I don't want to award a bounty if I feel that the user hasn't answered the question in the right fashion. How do I prevent it?
According to the SO FAQ
That is, you cannot prevent an answer from being automatically awarded. Note, this has nothing to do with the "Accepted Answer Checkmark", it simply means that the bounty will be auto-awarded after seven days to the highest voted answer that fulfills the criteria:
If an answer has been created and received at least two upvotes, then there is nothing you can do to prevent it from getting awarded (half) the bounty value.
EDIT As Andy E notes in his comment, you can downvote any answers which are at 2 votes and do not fulfill what you need, however you should leave a comment explaining why you do not approve of their answer, so that they can attempt to improve it. Remember that these people are volunteering their time to try to help you, so you should give them every chance to do that.
Remember, bounty can only be auto-awarded to questions
.. so you only need to worry about auto-award if these 2 conditions are met on your question.
If they are, and you're still worried about a "bad" answer getting accepted, answer the question yourself, and award the bounty to your own answer. Of course there will be no bounty award for this, but at least you'll have avoided an auto-accept -- if you're that worried about it.
If you don't want to award it to anyone do nothing and let it expire.
If none of the answers posted after you put up the bounty have at least 2 upvotes then no one will get the reputation (it will be lost), which I believe is what you want.
However, if at least 1 answer reached 2 upvotes, there is nothing you can do and half the amount will be awarded to the highest voted answer (counting only the ones posted after the bounty). You cannot prevent this but at least they get only half and not the full amount; better than nothing.