What happens if you put a bounty but there's no answer? Do you lose your rep? To whom?
I believe these two threads answer your question:
In short yes, the rep you put up for bounty will simply disappear into the ether if there are no eligible answers (meaning you didn't accept an answer and no answers have at least 2 upvotes). Sad but true :(
This answer provides links that explain how it works. To address the feeling that I think underlies your question: a bounty isn't a payment for an answer; it's a payment for advertising. If you pay to run an ad and get no sales that doesn't mean the place you ran the ad owes you a refund; similarly, a bounty that produces no answers is a sunk cost.
It's frustrating, I know; I've had bounties die on the vine like that too. I used to resent it, and then somebody explained the advertising model to me and I felt better. I hope it helps you too.
This is a very tough decision that I am sure was care fully thought through. The most obvious answer is this will prevent abuse of the bounty system. if I offer a bounty then people will pay more attention to my question and if all I have to do is not accept an answer to get my Rep back I am less likely to place bounties on my less important questions. Secondly, this will also help prevent users with large numbers of reputation from putting bounties on all their questions regardless of the relavence or importance. This will not stop abuse completely but it will limit users ability to open bounty after bounty knowing that they will just get it all back. Pretty much pay reputation too boost your really vital questions interest.