It sounds so easy: "Just keep it open." But there are inclusionists (keep them open) and there are exclusionists (close them). Your suggestion amounts to "just keep them open" while someone else might equally suggest to partially lock controversial questions against reopening.
See the problem?
Right now, if a question is open, you can only vote to close. If it's closed, you can only vote to open. Hence the yo yo effect.
The solution: Once someone votes to close, you have to capture the general consenus. Questions in controversy should provide both options: vote-to-open and vote-to-close. The total would be shown, as it is now. Once it reaches a total of (-5) it's closed. If the voting later reaches (+0) it is re-open.
Its an beautiful, simple, elegant solution. What am I am I missing?
Bonus: The system would no longer have to worry about aging votes (close votes eventually expire after a certain period of time). With the +/- votes shown, someone is free to counteract the random close-vote that accumulates over time.