I believe I've seen proposals to show a list of questions with close votes (to anyone with close privileges) that were rejected due to fears of dog-piling (can't find them right now however). I'd like to suggest something narrower though: showing said list on a per-tag basis, as deemed necessary by the admins.
I'd envision someone posting a question here on meta suggesting a tag for intensive cleanup, the usual discussion and voting, and then the admins enabling such a list for that tag temporarily if they feel it's warranted. Perhaps it could be another tab under
review, or a sidebar shown on questions with the targeted tag. Or if we're still really concerned about dog-piling, those questions could just be slipped back into the
review queue without indicating why.
A sidebar or review tab would make it a lot easier both to build the list, to get the right people looking at it, and to generally draw attention to these noble causes. With high-traffic tags that are popular with newcomers, this seems like the only way we can hope to really get things under control.
It's also got built-in rate limiting since it's tied to close votes, and if we want further protection against invalid closings, questions with the targeted tag which are closed during the intensive cleanup could also be put back into the review cue since we'd know which tag is being targeted.