Ideally, one of the voters would flag a moderator, but people are more reluctant to flag than to vote, and this requires that they somehow notice the question has been bountied.
Once a bounty has been offered on a question, if that question needs to be closed then a moderator needs to be involved.
Remember: closing is a nomination for deletion if the post is not edited to fix its issues. Saying, "Oh, no need to involve a moderator; if close votes don't expire, I'll just sit on my hands until the bounty is done and then close -> delete it" is effectively saying,
"I'm fine with someone putting serious effort into answering this question out of hope for a bounty reward, because I plan to delete the question and their answer, thus erasing both their effort and the reputation involved."
Don't do that. Contact a moderator - if the question must be closed or migrated, they can immediately cancel the bounty, removing the question from the Featured list and enabling others to close the question. If you're worried that the moderator reviewing your flag will not check for a bounty and just decline the flag thinking it doesn't require their intervention, clearly state in your flag that the question has an open bounty that needs to be canceled. The longer you wait, the more potential for grief exists for everyone involved.