This is part of a misunderstanding about the ways that bounties are supposed to work, and the way that bounties work on SO.
In real-life (or more specifically, movies, since I've never been a contract killer) the bounty is awarded upon completion of contract. If someone does not complete the task, then they are certainly not getting the money (and then usually something bad happens and action ensues.)
On StackOverflow, the bounty is to generate interest. You are not paying the bounty directly to the person who correctly answers your question in any real sense. The way that bounty works on SO is that you are 'paying' out a certain amount of reputation as an advertisement for your question in the hopes that people will answer it.
If answers are generated, and one of them is the answer to your question, then you accept it and there's no problem, the right person got the bounty. If none of the answers are correct, and none of them get above two votes then the bounty will go to no one and be lost. It is only if the answer gets more than two votes and is 'wrong' that you could worry about giving the bounty to the wrong person.
In that case, the best option is to engage the person who answered the question, if their answer isn't helpful to you, then add a comment to it explaining why. Also, if it becomes obvious that your question is unclear, then you should edit it for clarity. This has the added bonus of popping you to the top of the active list as well, which would generate extra interest.
Bottom Line: The bounty is to generate interest and answers for your question, it does not necessarily guarantee that one of those will be the right answer.