Since close votes expire during the bounty's duration, the current solution is to flag the post and get a mod involved. Why should mods need to be involved? We don't allow questions with bounties to receive close votes. It should go both ways.
For similar reasons, someone suggested that bounties should not be allowed on recently reopened questions. The answer was that:
The questions that, usually, go through multiple, swift, close/reopen cycles are those that are currently being discussed on meta or new ones, i.e. those that are not yet eligible for a bounty.
It then went into the "flag for mod" answer. I'm not familiar with other sites, but on Gaming, we close two day old questions all the time. If a question slips through the cracks and doesn't get closed right away, it is usually days or even weeks before the issue is brought back up.
The "flag for mod" answer works, but it seems unnecessarily complicated. What if a mod isn't around? It would be much simpler to simply disallow bounties on questions with close votes. Now that we can take close votes back, we don't even have to wait if a question with close votes gets fixed.