Shog9, Tim Post, and I met earlier today to discuss the issues with the close queue and we all agree that we need to take some steps towards making it more manageable and, more importantly, more reliable and useful than it is today.
Unfortunately, there isn't just one thing that we could do that'd obviously solve all of our problems, and there are some additional challenges in balancing this between Stack Overflow and the rest of the sites in the network.
Before I get into anything else, I should note that the goal of any improvements shouldn't be to just get the queue size down. If we just want to see a low number there, there are easy "fixes" we can do - inject fewer posts into the queue, reset the queue and start from 0, etc. The queue size isn't the point in and of itself. Stack Overflow gets a LOT of close votes and "recommend closure" flags every day, so the size of the queue is likely to always be rather substantial. What is important is that the posts that enter the queue get reviewed in reasonable time and that we can see some progress being made.
It is certainly true that there are more posts entering the queue than there are leaving the queue due to completed reviews, especially since we redirected "recommend closure" flags from the moderator flag queue to the review system. Some posts are removed from the queue because close votes just age out, but that's not always a good thing. If we can't trust that the posts in the queue are getting enough (or any) eyeballs on them, we can't necessarily age the votes correctly either.
We have considered raising the number of reviews available to people, likely scaled with the number of their reviews that resulted in the community-approved action ("leave open" on questions that didn't get closed, etc.). However, we want more people doing fewer reviews a day to spread the work around rather than just overload the already highly engaged reviewers to ensure that nobody burns out and that as many people get involved as possible.
The review limit is still on the table and we haven't ruled it out, but we feel that we have to make some more substantial improvements as well. We have a few ideas, some of which may be unworkable:
- more intelligent close vote aging that takes into account actions taken by users with close-voting privileges while they're looking at the questions with pending close votes or flags. (We are already working on this one. Expect a separate meta post once we've had a chance to vet and sanity-check the criteria we came up with against real data.)
- more aggressive dismissal or aging of "recommend closure" flags
- better advertising of the review feature to users who are able to vote to close.
- we can possibly do better at showing people close reviews in tags they care the most about. Maybe add some "these posts need your help!" kind of message to the community bulletin on a per-tag basis.
- split possible duplicates into their own queue with a better UI that'd make evaluating those flags easier. This doesn't so much fix the problem (beyond obviously lowering the number of items in the "main" queue), but it should make it easier to review those items.
- we prioritize newer posts in the queue, so some of the older ones are not getting seen. We could vary things up a bit to try and get older posts reviewed.
There are many other suggestions. Hat tip to Gnat and apaul34208 for summarizing the meta posts on this. For the moment, we are moving forward with improving the vote aging criteria. Once that's in place, we'll re-evaluate where we are and pick the next area to focus on.
