When you assign a bounty to a question, you are pretty much "paying" for extra attention. Your question is shown on a separate tab on the homepage, and on the pages of the tags with which your question is tagged.
The Stack Overflow community, by going to this tab and viewing your question, spends time away from the other questions on the site. This time spent can range from a few seconds, to several minutes, as some bountied questions require quite elaborate answers. This is time that could have been used to answer non-bountied questions.
Therefore, bounties aren't refunded if questions aren't answered because you are spending rep to gain attention, not answers. The fact that you may get an answer is different, and a result of the increased attention you get.
It does flow from user to user if you get a good answer. The point is that bounties take up the time of other users on the site, reducing the chances of a non bounty question getting an answer. If bounties were refunded in the case of no answer, everyone would be putting bounties on everything. Right now, the fact that you don't get that reputation back no matter what happens prevents users from using bounties if they do not absolutely need it. They'll often research their problem more and find solutions instead of putting up a bounty.