A question with a bounty has already been opened for at least 48 hours. In a busy community like StackOverflow, this is more than enough time to reasonably determine if the question is a good fit for the community. 

On a smaller site, like [Project Management SE](http://pm.stackexchange.com), moderators see every question that comes in; thus, they have a 48 hour window to figure out if they need to unilaterally intervene when the community cannot.

Once the bounty starts, as Shog9 mentions, there is a contract that exists between the asker and answerers that offers X amount of reputation in exchange for research and effort invested in providing the best possible answer. It makes sense during this time period to allow the question to remain open, at least for the duration of the bounty period.

*This is a temporary state. Once the bounty period ends and the answerers have an opportunity to receive their bounty award, the community can then take a hard look at the question and determine it's future.*

It's been established that any rep earned for answers with a least 3 votes will be preserved if a question is ultimately removed. Thus, if the same rule applies to bounties, then this shouldn't be a problem in terms of violating this bounty contract.