Today, the size of the close-votes queue is 55k questions and growing. Is that a problem?
Which action could be taken to clean it up? It seems to be growing faster than Stack Overflow members are capable to handle it.
closed as off-topic by gnat, Martijn Pieters, 3ventic, Emrakul, animuson♦ Apr 29 '14 at 23:04
This question appears to be off-topic. The users who voted to close gave this specific reason:
TL;DR Educate active reviewers to filter SO Close Votes queue by single close reason, with "duplicate" and "too localized" additionally filtered by their favorite tags.
Based on my experience, this leads to substantial productivity gain.
Boring details on what led me to above conclusion are provided below; feel free to skip these.
Initially I've been going through close votes queue unfiltered. This worked just fine in all other queues I tried (dozen or two queues at SO and some smaller sites) and I saw no reasons to do it other way.
This queue felt a bit difficult from the beginning, but I thought I'll get used to it after a while. 300+ review actions passed, but it still felt cumbersome, at this moment I got curious why is that?
My testing of filtering by close reasons began with "off topic". Bingo! Review went smooth and fast, and most important, I felt comfortable - just about like I felt with all other queues in new system. I tried "not constructive", then "not a real question" - great, the magic is there.
Testing "too localized" though didn't go so well. It felt cumbersome again - just the same feeling of having to deal with too many distractions. This lead me to idea to try filtering it further. I added a tag I was comfortable with and the magic flow was back again.
Based on my experience with "too localized", I switched to testing "duplicates" filtered by favorite tag from the very beginning. This worked well, too. Well, to be precise, duplicates review went slower (much slower) than the rest: I had to study both questions, and in some cases I also had to check the answers to both questions. But (and this is very big BUT) what is important, it felt really comfortable - as opposed to my prior experience with unfiltered queue.
To reduce the backlog:
define a function: close(p) = ....
where the right hand side includes the number of close votes, the total rep of all the close voters, the net vote score of the post, the number of answers to the post, and the age of the post.
In other words, 3 close votes from high-rep users, on a negative-score question, with no answers, should just close the thing. Or some rule like that. Then we could concentrate on content that had some chance of having value.
Take a minute here for a sanity check:
Update as of 10/24/2013: 86k need review, 1,151k all-time reviews
There's a massive amount of reviewing being done there. It takes 5 voters to close or 3 "do not close" reviewers to kick a question out of the queue without closing. And yet, folks are making some impressive progress.
Frankly, I don't particularly think this needs to be sped up any - closing is hard by-design. If it takes a while to get through the backlog, that's probably ok...
Update: review task creation / deletion over time
In response to LittleBobbyTables' concern, here are some graphs that may help to illustrate the problem with clearing this queue:
Close review tasks created
Close review tasks deleted (either completed in review, closed or deleted elsewhere)
Review tasks are created in response to close votes or flags - these have been going up.
At the moment we can filter by tag and by close reason, but off-topic has 7 sub-close-reasons.
It may improve the workflow a bit to add additional filtering and be able to narrow it down to one or two sub-close-reasons.
After all this time, I wanted to add a view of why people don't visit the queue -- or -- at least, why i don't visit the queue.
I have a limited number of close votes. I visit the queue, I use them all, and I feel like I'm the bellhop in Hilbert's Hotel. The amount of work to do remains infinite. I have zero feeling of accomplishment.
In my view, the size of the queue is a view into the ongoing torrent of crap dumped onto the site. I'm willing to keep trying to help deal with it, but I need a larger gun if you want me to help shoot this many zombies.
Just as an input to this discussion, a small progress report.
After a couple of weeks (or maybe more? don't have the energy to check) on Stack Overflow spent maxing out my close vote queue reviews filtered by the cocoa and objective-c tag, today I got the very nice message
(thanks to @gnat above for attracting my attention to the filter)
A couple of random observations from this marathon clicking exercise, in no particular order:
My suggestion: Encourage people who create close votes to also do some reviewing.
When someone votes to close a question, I suggest that the system should check their reviewing records; if they haven't reviewed any close votes recently, then a polite notice should be shown asking if they would consider doing some reviews to shorten the backlog as well as just adding to it.
I would hope that this would encourage more people to do a few reviews who wouldn't do so otherwise.
(the above assumes that the user has permissions to do reviews, of course)
I see this working a bit like the voting system does now. Right now, if you mainly vote on answers, you might occasionally get a message when you vote on one saying "You haven't voted on a question lately. They also need some love." (or words to that effect).
So how about doing the same with close votes -- if someone's issued a number of close votes without using the review system, they could get a gentle reminder like "You haven't done any reviewing lately. Please help reduce the workload by reviewing some questions with close votes."
It's not intrusive, it only pops up once in a while, and it doesn't force anyone to do anything, but it might just help do something about the backlog.
(by the way, the close vote review queue is growing rapidly at the moment; it's pushing toward 70k. It's gotta be seen as a problem sooner or later, hasn't it?)
Let's take a look at the problem. As of today, this queue is up to 81.8k questions in the queue. Assuming each has 1 closed vote, that means that a total of 320K actions are required to close all of the questions. Given 20 per day, that means that 160K people would have to clear out 20 in order to close the queue. Then after it is closed, then more people need to review them than have been doing so. So, how can these get reviewed? There are a few ways:
Okay, given these options, how could any of them be done?
Increase number of reviewers?
Close vote reviewing is a painful, thankless task. Sure, it's the easiest badge of the various custodial badges to earn, but a closed vote decision can be quite difficult sometimes, depending on the nature of the question. Perhaps there should be some incentive to encourage more reviewers, either in the form of reputation, or increased privileges.
Decrease the number of things to review
There are a couple of ideas that I have that fit into this category. Either somehow things need to be automated better, or the number of close votes could be reduced. As the latter could be dangerous, perhaps we should just allow certain users to have more closed votes, perhaps those with a very high reputation, or maybe those who have reviewed more closed votes overall. Both help to filter out the relatively new closed voters, and leave those with more knowledge of how the site goes more power to help it out.
Increase the number of things that can be reviewed
This improves the situation so long as you don't burn people out. As the pile is getting deeper, it could be a quick short term solution.
Decrease time per review
Filtering by tag or closed reasons, better displays to help users figure out what's going on, anything to make the task of reviewing easier increases the likelihood that a user will use all of their closed votes per day, and come back tomorrow and use them too.
I'm sure there are other ideas out there, but here's my preferred method:
Disclaimers: Though I am relatively new as a registered user of the site, I have been using it for a long time and have read a lot of posts that have been closed for various reasons. Also, since I can't yet participate in close-queue reviewing, I certainly can't speak of how easy or hard it is to do so.
Most propositions made here discuss how to deal with the close-queue as it is now. But it is even important to reduce the number of questions that do enter the queue in order to eventually reach the end and then keep up without tremendous efforts.
Now, a quick look at recent questions that were voted to close shows that a large number of these questions come from users with no reputation or almost none. A person learning a new language or technology, or even learning programming for the very first time, either at school or by following whatever tutorial found somewhere, is certainly the archetype of these users.
I don't think it is pertinent to discuss if these persons are being too lazy. It is obvious, to us at least, that the answer they need is lying somewhere around. But when you know very little, it might be hard to decode answers that are not tied exactly to your situation. So they come here, expecting to receive help. They are not considering the fact that their questions should provide common, persistent knowledge for others. And even if they would consider that aspect, their view of "knowledge worth sharing" is certainly tainted by the fact that they indeed have a very limited knowledge of the technology being learnt. Is it bad that they come asking for help? I don't think so. But their questions and the associated answers are certainly not assets to the Stack Overflow community.
So how does all that might help the present discussion? It basically tells us where to put efforts in order to reduce the influx of to-be-closed questions. These persons are looking for answers cut precisely to their needs, which they won't find anywhere. Stack Overflow is very popular and very well known for offering quick answers to questions being asked. Add those two facts, and you realize that you can't expect these users to stop coming here, nor can you expect them to stop asking low-quality questions. Not providing answers to these questions will not discourage students from coming here. But having a few persons trying to provide a quality answer (the kind that guide the user toward better understanding, rather than the kind that is to be copy-pasted), while five persons are reviewing the very same question to vote a close is certainly a waste of everyone's time.
So here are my suggestions:
Well, that's it for now. Hope that helps.
Another possible solution might be to bring questions that have more votes to the front of the queue. In other words, if you have 4 close or 2 leave open votes on a question, if it gets brought to the front of the queue, it is more likely to be resolved quickly.
... or, you know, they could just give us more close votes.
It's 9 AM here, and I'm already out of close votes for the day. On most days, fifty votes just isn't enough to deal with new questions plus questions in the Close Vote Queue.
I don't need a prize, or a badge, or even thanks... I just want more freaking votes.
Automatically prioritize posts in the review queue that are tagged with either the users favorite tags or tags selected by the user. This could be limited to e.g. 3 tags if performance is a problem.
If there a no more posts with the specified tags, show other posts (without having to remove the "filter", which is just an annoying UX).
I get the impression that except perhaps for test questions we're mainly getting recent questions to review. I suggest that once you've handled the current maximum 40 questions in a day (or perhaps earlier) you start processing the from the back of the queue, or just generally give us more old questions to process, and probably a higher limit per day in response.
There are currently 95.3k items in the Close Vote queue, and 2 items in the Reopen Votes queue.
The number of reviews needed to close a question from the CV queue is 5, which is ridiculously high. Here's a stat:
That's 73% accuracy from <3k users flagging; i.e. the people we believe to be least-able to recognise a close-worthly question are already getting 3/4 of them right... do we really need a further 5 >3k users to back them up?
Here's my actual feature-request:
Let's find a number
If we require only 3 reviews instead of 5, and we end up with the same outcome 95/90% of the time as we would do using 5 reviewers, why aren't we doing that already? Yes, I just made those numbers up. Shog help me..
For those that are dubious as to whether this would open the floor for questions wrongly getting closed, let me calm you down with this;
If you're still struggling, then how about I knock you clean out by suggesting we add the ability to flag a question for reopening. This would force the question into the reopen queue, and handle the small percentage of questions that incorrectly get closed. It would also allieviate a proportion of the flags moderators have to deal with at the moment. If we're worried about the question-owner spamming the flagging option, only enable it for non-owners.
I have another suggestion to make:
Let vote-to-close flags count as silent close votes and convert them to real ones once they have been validated by a user actually voting to close.
Once a VTC-flag gets raised on a question the question enters the review queue. Now the post needs five votes by five different reviewers to be actually closed. Lets say a question enters the queue with two flags raised. A user reviews the question and votes to close. After this voting the close vote count of the question would actually be three, because the two flags cast have been converted to actual votes.
Consequently spoken: When a question has an active close vote (one that has not aged away) a new VTC-flag cast on that question should automaticalle be converted to a close vote and be validated.
To make a long story short: Once a user with the necessary privileges has decided to cast a close vote on a question, let others (whether they have the closing privilege or not) jump on the train by flagging the post.
Allow a metric other than filtering by tag. Such as searching with a text box or sorting by date or sorting by OP reputation.
It doesn't make sense to constantly be reviewing content that was posted 49 minutes ago and is still being edited.
Here are my thoughts (posted under another similar question):
And some more suggestions:
Age all posts out of the Close queue after two weeks. The rationale is that if five close votes cannot be accumulated against a post in two weeks, closing the post is likely a dubious decision anyways. Additionally, the close votes against those posts would also be cleared, enabling users if necessary to attempt a reclose of the question, without having to wait for an edit.
Optionally: As a corollary, one might argue that the same criteria be applied to the re-open queue: age out all posts two weeks after an initial re-open vote. However this queue seems to be unmanageable, so symmetry may be unnecessary.
This would not interfere in any way with the ability of the community to rapidly close (and if necessary delete) posts that absolutely must be dealt with quickly. In fact, this proposal might assist in having those posts stand out, enabling them to be dealt with more efficiently.
The immediate effect would be to clear a large number of posts from the Close queue, and with the aging mechanism in place, hopefully allowing it to be maintained at a reasonable size. As mentioned by others, the advantage of seeing a manageable Close queue is increased community interest in maintaining it at a manageable size, presumably allowing more poor quality posts to be closed, more promptly.
If this were to be undertaken on a trial basis, what is the worst that would happen? 74,000 or so pots (less those nominated for closing within the past two weeks) would be cleared. Most of those posts weren't going to be reviewed in less than a few months anyways.
If this change is adopted and becomes successful successful, one could imagine creating a separate long term Disposal-Review queue with the aim of slowly working through and assessing old posts. If they were sorted inversely on a quality measurement, the likely worst offenders (or at least, best candidates for action) could be dealt with soonest.
update: The underlying assumption behind this proposal is that the Close Queue is broken, and needs fixing; that a mechanism which worked well for several years has now been overwhelmed by the site's popularity, and needs refreshing.
From my answer here:
I think a contest is the right approach, but it needs to be done in a big way.
First, set an amount of time, I was thinking two months, though 45 days is fine. If you do that starting next month, you can end things at the end of the year, but that's not crucial.
Second, you give out one-time permanent badges for the contest commemorating their achievement forever. Let's say for 250, 500 and 1000 closes in that timeframe (obviously you should probably remove the review limit for the duration of the contest).
Third, you give out either rep or even specialer one-time ranked permanent badges for the top 3 closers.
Make it as big as the Winter Bash Hat thing, go wild.
Would it not be possible to change the minimum amount of required rep to 2k instead of 3k? I think 2k is already quite hard to get, and proves that you know your way around SO and should be capable of deciding if a question needs to be closed.
More reviewers obviously helps trim down the queue.
Should we make the minimum reputation level higher for asking questions? This would significantly lower the amount of bad questions that are flooding the queue.
Most of the bad questions are not a result of users being stupid, just lazy. If they would be forced to participate in the community for some time, before being allowed to ask a question it would result in two things:
I've made a little research to find out how many "good" questions would be lost, if there was a bar. The procedure was as follows:
1) I've queried SO database to retrieve all closed questions since
2) I've filtered the closed questions to get "questions from fresh users". A "question from fresh user" is a question with creation date distant no more than 5 days from user's account creation date.
3) I've accumulated all user IDs from the previous step, to produce a list of "users who asked a question within
4) I counted the "good" questions created by these users. "Good question" is a question that is not closed, and has score at least
The results are:
1) There were
3) These "fresh" questions were asked by
If we define "good" question as a question with score
If it would be possible to filter out users who asked a "bad" question not using a global rep bar (for example: introduce rep bar after first asked question was downvoted heavily or closed) than only
(*) it's "probably" because it is still possible that these questions would have been asked anyway, after the user would reach the bar
(note the throttling - you may need to run the query more than once, uncommenting the
(note the throttling - you may need to run the query more than once, uncommenting the
At time of writing, there are 120K+ close vote questions.
Seeing as something drastic needs to be done, there need to be more harsh actions taken, which could looks as follows:
By way of example, for point 2, if you helped close 20 questions by downvoting it, you get 20 extra reputation, which is not exhorbitant. It's just enough to say thanks, but not so much that people will start closing questions that are actually valuable.
You should perhaps then increase the reputation required to get the downvoting privilege, because to someone with 125, downvoting will look like an easy way to gain reputation.
Sure, you don't get all your stats about why stuff was closed, but things are too desperate now to worry about that.
To reduce the backlog:
Gamify closing more.
Right now people review the close votes queue to get badges and for the love of the site. If that's not sufficient to clear the queue, add more incentives. SE is an excellent example of how well gamification can drive participation in an otherwise not very exciting activity. Why not apply the same principle here?
Obviously this will require some careful thinking to avoid incentivizing undesirable behavior, but that's the sort of tinkering that SE requires generally so it's nothing new.
One way to gamify the close queue without incentivizing bad behavior. This is meant more as a proof of concept than a completely foolproof system, something to distinguish between incentivizing well and incentivizing poorly.
Each close vote cast earns 1 rep for the caster if the question is subsequently closed. Each Do Not Close vote earns 1 rep for the caster if the question reaches five DNC votes before reaching five close votes (i.e., being closed). If a caster is on the wrong side of the majority, voting DNC on a question that everyone else votes to close, the caster loses a point. Reviewers aren't shown others' votes to avoid blindly voting with the majority.
In this system, there's a reward for reviewing questions that should potentially be closed, but only if other users agree with you. This is meant to be exactly analogous to the general SE model in which questions and answers are rewarded if and only if the community judges them useful.