Yes, I think if you don't get any answers, you absolutely should get your rep back; you've paid for a service that wasn't delivered - that's a refund in anybody's book.
I asked a question (and I don't think it was a crap one - quite clear, but it perhaps is a bit tricky) that had very low views for a few days and so I've whacked a bounty on it.
Still, as yet, no answers, and the number of views hasn't gone up.
If after the 7 days I still get no answers, then I'm going to end up paying with rep because the community could not help. So the SO message is "If your question is really difficult, and you incentivise the community to answer, but they can't, then you'll be penalised".
That roughly equates to "please ask only questions that somebody here knows the answers to".
If this Q of mine ends up getting a good answer now, then of course my comments here will be less relevant I suppose; however for those people that do ask good, bountied, questions but which nobody knows the answer to are being treated unfairly I think.