What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

This question already has an answer here:

There are (at the time of writing this; see edits at bottom of post) 54.2k questions with close votes on the review page on Stack Overflow. Unlike the other review tasks, which all usually return to 0 quickly, the number of questions with close votes continues to rise. According the this Meta Stack Overflow post, there were 51.8k close votes as of April 25 2013. In the previous five weeks there has been a deficit of about 400 votes.

Isn't there something that should be done about the close vote system?

As opposed to related question Why is the close vote review queue so full?, which was about why is the queue so full, this question is about the growing queue and what can be done about it.

It's a graph!

Date                Questions in close vote queue
2012-08-12           55000
2012-08-19           54900
2013-03-03           49300
2013-04-25           51900
2013-06-12           54600
2013-06-14           54800
2013-07-04           57900
2013-07-06           58200
2013-07-11           59700
2013-07-19           61600
2013-08-02           66100
2013-08-20           70400
2013-08-30           73800
2013-09-02           74100
2013-09-10           76500
2013-09-18           79600
2013-09-19           80000
2013-09-30           82100
2013-10-02           82800
2013-10-24           86400
2013-11-12           91900
2013-11-13           92900
2013-11-14           93500
2013-11-27           96900
2013-12-03          100100
2013-12-04          100500
2013-12-12          105400
2013-12-13          106000
2013-12-17          102300
2013-12-18          100700
2013-12-20          100100
2014-01-09          103800
2014-01-10          104400
2014-01-13          104600
2014-02-03          113900
2014-02-25          121300

Using the above data:

Average daily increase in vote queue per day (as of 2013-12-03): 94

Average disregarding the two outliers (two data points from '12): 184

share|improve this question

marked as duplicate by ben is uǝq backwards, Lucifer, Hugo Dozois, gnat, ChrisF Jun 17 '13 at 21:22

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

3  
It has been even below 50k. 47k is think. –  juergen d Jun 10 '13 at 13:41
4  
Close votes has never reached 0, and honestly it most likely will never reach it. People differ in their opinions on questions; therefore, there will always be questions in the close votes queue. There are questions that will only get a few votes and will eventually drop out of the queues, but with the high quantity of bad questions being introduced daily there will always be work to do in this area. –  Josh Mein Jun 10 '13 at 13:44
12  
@JoshMein: But it seems the number is rising instead of droping. –  juergen d Jun 10 '13 at 13:47
3  
@juergend That is a testimate of the mass low quality questions that are introduced daily. This is why quality rules are being increased to try to prevent this problem. Honestly, I dont think this queue should ever hit 0. Not all questions are cut and dry on whether they should be closed. –  Josh Mein Jun 10 '13 at 13:49
4  
@JoshMein: Maybe it will or should never be 0. But the problem here is that it seems to hit 100k or more some day. THAT is the problem. –  juergen d Jun 10 '13 at 13:51
3  
@juergend I would honestly like to know how many people frequent the close votes queue. It is the queue that requires the most thought and time to do the job correctly. Maybe people just dont feel like dealing with it. I know I dont sometimes. –  Josh Mein Jun 10 '13 at 13:55
11  
How is this question a duplicate of my question? My question was why is the queue so full. This question is about the growing queue and what can be done about it. –  juergen d Jun 10 '13 at 17:01
5  
@juergend I don't think it is either. –  Mooseman Jun 10 '13 at 17:03
7  
I wonder why each question requires 5 CVs. Could the number of required CVs not be correlated with the quality score? Some questions are so blatantly bad that anyone (except the OP) would close it. Three votes would suffice. –  Gert Arnold Jun 12 '13 at 10:11
12  
For me the problem is also that I don't like doing things that just don't matter. When I spend half an hour on the close votes queue, it's still at the same number, sometimes higher. So yep, I've kinda given up on it, and focus my energy on queues that actually still give that sense of satisfaction of a job well done when it says 'No more items to review'. Close queue needs to be changed - some kind of mass autoclosing algorithms, and/or less votes to close. I'd think lowering the number of close votes required by 1 for every negative score point, minimum 2, would help a lot. –  Niels Keurentjes Jun 12 '13 at 14:41
4  
Today, for 43 glorious minutes. –  George Cummins Jun 25 '13 at 21:54
3  
In the past two-and-a-half months, the queue has increased by nearly 40%... –  Mooseman Aug 30 '13 at 11:29
3  
I graphed it. –  michaelb958 Sep 2 '13 at 22:50
2  
@DiegoCNascimento According to stackexchange.com/leagues/1/alltime/stackoverflow, 17k users have 3k+ rep. Should be enough, if they're all willing to review well. –  Mooseman Sep 18 '13 at 23:59
3  
To infinity and beyond! –  LittleBobbyTables Nov 12 '13 at 15:36
show 25 more comments

5 Answers 5

The review queue lengths express the community's collective opinion on the work asked from us.

Some queues are always near zero:

  • First Posts
  • Late Answers
  • Low Quality Posts
  • Suggested Edits
  • Reopen Votes

This has to mean that the community thinks it is all right to chip in a little work to help reviewing these things. The workload is obviously acceptable, and so are the various conditions for each queue: Some only require a single review, others more, and the work required is reasonable, ranging from a click (a vote) to editing if needed/worthwhile.

The close vote queue on the other hand, is a proposition that the community is turning down. In the beginning you could say that it was huge because of all the old close votes hanging around in the system. Now, however, we have numerical evidence that it is growing on a daily basis, and can conclude that it is simply not handled in any meaningful way by the community, compared to the other queues.

I can see no other explanation than the community is rejecting the deal on the close vote queue. If we really wanted to go through it, we could. But we don't go through it, so it must mean that we somehow collectively don't think it is right.

My own personal analysis of the problem is that the close vote queue represents a grossly uneven deal between Stack Exchange, the company, on the one side, and the community, us, on the other side.

The imbalance I see is this: In a typical scenario a likely not very experienced user asks an unresearched, badly formulated and unfocussed question. He (or she) has spent very little time and almost certainly no effort in asking the question.

What is expected of the community under the deal with Stack Exchange is: Five of us spend time and effort to try to understand what the poster is asking, and, at least in the case of a dupe, research the topic for the OP and then classify the problem in five different categories.

Personally, I don't think this is a square deal. Not only are we asked to do the poster's work for them, but we're supposed to produce a whole team (five people) to do it, each one independently - and cross check the other reviewers' work in the process.

That simply doesn't sound like a reasonable proposition, and so the queue is left to slowly grow as questions get a single flag or vote, which nobody cares to decide on.

share|improve this answer
21  
I would also point out that reviewing CV's requires 3k rep (the most out of any queue). –  mikeTheLiar Jun 10 '13 at 19:17
    
@mikeTheLiar That's true, I wonder if the difference in the number of active participants at the various rep levels can explain the difference. My instincts tell me it is not enough to explain it, but I may be wrong. –  Monolo Jun 10 '13 at 19:27
3  
If what you suggest is true, then every site's review queue would be growing. But they're not; it's just SO (as far as I know). It's far more likely that seeing a huge number like that is just demoralizing, so nobody wants to help slowly chip away at it. –  Nicol Bolas Jun 12 '13 at 7:07
2  
You also need to remember that some people are viewing old questions and voting to close them because standards have changed over the years. –  Josh Mein Jun 12 '13 at 15:43
    
minor nitpick: the company is Stack Exchange; they work on Stackoverflow –  Amelia Jun 12 '13 at 17:02
3  
@NicolBolas I agree with you - the expectation from the SE staff that we go through that huge backlog of votes is obviously not accepted by the community. Whether it is because it is perceived as demoralising or an unjust proposition is probably a detail. –  Monolo Jun 12 '13 at 17:36
1  
@Hiroto Thanks, corrected. –  Monolo Jun 12 '13 at 17:40
8  
terrific analysis. "Five of us spend time and effort to try to understand what the poster is asking, and, at least in the case of a dupe, research the topic for the OP and then classify the problem in five different categories. Personally, I don't think this is a square deal..." -- so true –  gnat Sep 19 '13 at 12:40
2  
@gnat thanks for digging out this passage, which is the core of my argument. In true over-analytical style I have managed to bury it way down the post! –  Monolo Sep 19 '13 at 13:22
1  
"true over-analytical" -- as an author, you're expected to be humble about what you wrote, but I am not obliged and I can tell, the rest of your answer is very well worth studying, too. I think I understand why it doesn't score high (yes, 33 upvotes for quality content like that is not high) - truly profound thinking typically doesn't mix well with crash boom bang stuff that attracts hundreds upvotes. But this didn't stop me from getting a major pleasure studying what you wrote –  gnat Sep 19 '13 at 20:50
add comment

Maybe we should reconsider the requirements to moderate that queue.

Is the difference in experience between a 3K and a 2K reputation user (or even a 1K user) really distinguishable enough to warrant separate access levels for these queues? Can we just lower the bar of entry?

Perhaps tenure, or activity considering lurkers, could factor into review queue rights.

On a discussion based community typically such a task would be performed by vetted moderators, as proven knowledge of the rules is required for the task.

Could we allow lower rep users to vote on a series of close requests and then, as we have the community vet posts, I would suggest we review a sample of their decisions to see if they are on target with Stack Overflow's standards?

If they passed approval allow them access to the Close Votes queue without moderation - relying instead on the existing system of requiring 5 votes.

Obviously we would want to restrict such a system by reputation/activity to some degree to avoid new users wasting peoples time - also this would require another queue, "Close Vote Approvals" or something, available to higher rep users.

Just spitballing.

TLDR; Mor dakka.

share|improve this answer
5  
I'm not sure there's such a large number of users between 2k and 3k rep that would be willing to review close votes (and that would do it properly) to actually make a dent. Also introducing another review queue to review those people seems like it would simply increase the work load. –  Anthony Grist Jun 12 '13 at 10:15
    
Yes I definitely agree that adding more queues = more work, however the benefits may outweigh it. The limitations of applying for review would need some careful consideration as it would affect this. I would also suggest that if a user was approved that their votes for the review be processed, as well as the reviewers, and put into the system as actual close votes. –  Amicable Jun 12 '13 at 10:28
    
In order to clear out the close votes queue we'd need a substantial increase in the amount of people reviewing it. If a large proportion of those people require reviewing in a new queue then we've simply replaced one unmanageably large queue with another one. I just don't see any benefits, since the people reviewing the new queue are likely the same people who are currently reviewing the close votes queue. –  Anthony Grist Jun 12 '13 at 10:45
    
I don't follow your train of thought. I do not see this extra queue as being unmanageable or a replacement for the existing queue. –  Amicable Jun 12 '13 at 11:03
    
I am not suggesting that lower reputation users require a higher rep user to vet their votes forever, that would have no benefit ever. I am suggesting they be vetted just once for approval to access the main queue. The benefits as I see it would be an increase in the amount of users available to work on the actual Close Votes queue. –  Amicable Jun 12 '13 at 11:05
1  
According to stackexchange.com/leagues/1/alltime/stackoverflow, the number of users with a number of reputation between 2k and 3k is 7978. –  Mooseman Jun 12 '13 at 11:19
    
@Mooseman: So not exactly the largest pool increase. There is 28,580 between 1k and 3k. –  Amicable Jun 12 '13 at 11:29
    
@Amicable Right, but I agree with Anthony that lower rep users may not do it properly, which poses a risk toward the entire site. –  Mooseman Jun 12 '13 at 11:43
    
Agreed, which is why I suggested a review queue. –  Amicable Jun 12 '13 at 11:53
add comment

Why not have a close vote reviewing event?

  • Raise the 40 reviews/day to 50
  • If you finish the 50 reviews, you get an event-specific bronze badge.
    • Finishing the 50 reviews for 5 consecutive days gets you the silver event-specific badge
    • 10 consecutive days gets you the gold event-specific badge
  • 3 failed audits during the event disqualifies you for 4 hours from reviewing close votes
  • Until the close votes reach 0.

After the queue is cleared, change the 'Steward' so it can be earned multiple times for close vote reviews. (e.g., 1k, 2k, 3k, etc.)

share|improve this answer
3  
If you make changes to get reduce the queue down to 0 why revert the changes when it gets there? Won't that cause the queue to just start increasing again and get it back to the original problem again? –  Joe W Jun 12 '13 at 14:06
4  
I like the proposal except for the failed audits exclusion duration. The audit system is way too flakey to rely on it so heavily - I've also failed audits in the past on which I still disagree with the 'requested outcome'. It's rare, but happens. I'd say disqualify for 1 to 4 hours after 3 audits, just to say "go take a rest and start anew when fresh". –  Niels Keurentjes Jun 12 '13 at 14:45
    
Make it harsher, yet more lenient: any time you fail an audit, you are unable to review for the next hour. –  JoshDM Jun 12 '13 at 15:10
    
I edited the post: "3 failed audits disqualifies you for 4 hours" and the action for after the queue is cleared. –  Mooseman Jun 12 '13 at 16:54
6  
Don't introduce more badges here - it is going to draw in more robo-reviewing activity, and we have enough of that already. And reducing the threshold for audit ban is just giving robo-reviewer a chance to run wild. –  nhahtdh Jun 12 '13 at 17:03
    
@nhahtdh Is this case, there is no right answer. Some people will always be robo-reviewing. –  Mooseman Jun 12 '13 at 17:04
    
Is robo-reviewing on that queue actually a problem? Because I sort of doubt it. –  Nathaniel Ford Aug 23 '13 at 17:22
    
@NathanielFord Robo-reviewing is always bad. However, if there are already four agreeing close votes, it's extremely unlikely that all four are wrong. –  Mooseman Aug 23 '13 at 19:15
    
@Mooseman By 'always bad' do you mean it happens a lot, or that the act of robo-reviewing is always bad? I meant to question what the incidence of roboreviews are; if they're very low it's not worth our time. –  Nathaniel Ford Aug 23 '13 at 19:17
    
@NathanielFord - Yes, it is. There have been a number of good questions that were closed because of four blind close votes from the queue before audits were put into place there. People are so fixated on reducing the number in the close vote queue that even good reviewers elsewhere were voting to close 100% of what they saw in that queue just to blaze through it. That's harmful to the community, too, because it has angered a lot of people who took the time to ask good questions, yet were rejected anyway. –  Brad Larson Sep 2 '13 at 16:36
add comment

As I understand it, Most questions that make it into the close vote queue actually end up getting removed from that queue.

I don't think that we need to encourage a massive amount of reviews so much as work on the margin between posts coming in and posts going out.

There are a few things that I can think of to deal with this

  • Allow unlimited reviews for people who have already earned the gold badge.
  • Give high-rep users more weighty close/leave-open votes.
  • Make the queue more conspicuous so that more people find the review queue.
  • You can have a criteria to remove questions from the close-votes queue(for instance, if it has an and accepted answer with a certain threshold of upvotes)

You can also introduce another gold badge for reviewing some 5000 or so reviews. This would probably be the most effective option, but it would also attract the most badge grinders.

the point is that we want to whittle the number of extra reviews per day to 0, and you really shouldn't need more than 500 extra reviews per day for this.

share|improve this answer
2  
Allow unlimited reviews for people who have already earned the gold badge. Gold badge != Quality review. Give high-rep users more weighty close/leave-open votes. This must be introduced outside the queue (globally), not just inside the queue (it will be inconsistent otherwise). Make the queue more conspicuous ... Probably OK You can have a criteria to remove questions from the close-votes queue Can't say leave open vote is good enough, but insta-remove is not a good idea. –  nhahtdh Jun 13 '13 at 3:54
1  
@nhahtdh If someone already has a gold badge, they're definitely not a grinder, so you'll get less robo-reviews that way –  Sam I am Jun 13 '13 at 14:17
    
I would like to see some statistic, but it is very likely that is true... Still, I don't like the idea of unlimited review. 40 reviews per day is quite a lot if you do it seriously. –  nhahtdh Jun 13 '13 at 14:24
1  
@nhahtdh If someone already has the gold badge What do you suppose they would be grinding towards? –  Sam I am Jun 13 '13 at 14:28
1  
The top review list. I did suggest in another question: meta.stackexchange.com/questions/170586/… to tone down on the gamification of reviewing feature. –  nhahtdh Jun 13 '13 at 14:30
    
@SamIam: The daily limits predate robo reviewers and the review queues. "Not a robo reviewer" isn't good enough for unlimited review privs. –  Manishearth Jun 14 '13 at 11:45
    
I'd defiantly want to have another badge to work towards. –  Undo Jul 7 '13 at 4:05
add comment

I think the best thing to do about it is to clear all questions from the queue as a one-time thing. Once the queue is down to a reasonable size, you'll be more likely to get people to actually pay attention to it.

Granted, that means that there will be 50k+ closed-voted questions that won't be reviewed. But at least new ones will be more quickly dealt with.

The purpose of the close queue is to draw attention to bad questions in tags that don't draw as much attention. In high-traffic tags, bad questions are usually shut down within 5 minutes. In low-traffic tags, they are not.

share|improve this answer
1  
That won't really solve anything as it will just slowly grow over time and the same issue will happen again just sometime down the road. –  Joe W Jun 12 '13 at 18:44
4  
@JoeW: That depends on why it's currently growing now. It's a lot harder to convince people to help chip away at a 50+k list than it is at a 500 question list. –  Nicol Bolas Jun 12 '13 at 19:23
8  
I think it is reasonable to do the same thing as in the past, where a number of posts are hidden away and reintroduced as the queue hit 0. –  nhahtdh Jun 13 '13 at 3:50
1  
@nhahtdh That seems like a very good idea. We could then know for sure if the problem is that having a 50k list is demotivating. –  Pablo Jul 6 '13 at 13:06
1  
@nhahtdh, that's probably worth its own feature request. –  Benjol Sep 11 '13 at 9:49
add comment

Not the answer you're looking for? Browse other questions tagged .