First of all, 3 days out is not "about to expire," there is a lot more action on bountied questions in the last day and even the last couple hours of the bounty, as it works its way up the "featured" list.
Yes, your 50 pts are gone no matter what. "Unresolved" bounties are lost; see here: What happens if there are no answers to a bounty question?
If someone posts a non-answer just for the bounty, flag it and I'm sure the mods will delete it (deleted questions aren't eligible for bounty).
If you get an inadequate answer, either you can let it take the bounty as it was the best advice offered, or you can answer your own question and award that the bounty (you won't get the rep back, but you will prevent it from being automatically awarded to an insufficient answer). (apparently this was recently removed)