According to the description in the help center. the bounty is only auto-awarded if there exists an answer that was created after the bounty started with a "minimum score of 2". Quote:
If you do not award your bounty within 7 days (plus the grace period), the highest voted answer created after the bounty started with a minimum score of 2 will be awarded half the bounty amount. [...] If there's no answer meeting those criteria, the bounty is not awarded to anyone.
This means that if the voting mechanism works the way it is supposed to, downright bad or irrelevant answers would not get a bounty, as they would not have a score of 2 or more.
However, I have noticed that bad answers (at least bad IMHO) sometimes receives a lot of upvotes, so the voting mechanism is far from perfect.
It is also a fact that people set bounties, receive a useful answer, and then don't bother (or forget) to accept the answer or award the bounty. I've experienced more than once that the person who set the bounty has thanked me for the answer in a comment, saying that is solved their problem, but never returned to SE in time to accept the answer or award the bounty. When I experience this, I am happy that the auto-award mechanism at least gives me half the points for my efforts.
So I think that auto-awarding should still be the default. Because of the "temptation" of the bounty, people tend to spend more effort to solve a problem with a bounty, and the present auto-award mechanism solves the annoying problem of bounty-setters who don't care what happens after their problem has been solved.
Also, there should no way to get the bounty back, as such a mechanism is open to obvious abuse.
However, there should be a way for the bounty-setter to prevent the bounty from being awarded, even if the highest voted answer has a score of two or more.
Proposal: Add the criterion that if an answer has been downvoted by the bounty-setter, then it is not auto-awarded the bounty, no matter what its accumulated score is.
This gives the bounty-setter a way to prevent the bounty from being awarded a bad answer, while still allowing good answers to receive an auto-award if the bounty-setter is just negligent.