What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

This question is related to this (jan23), this (mar 12) and many others; with the difference that close queue on SO today is 20k longer = 74k.

I read & understand about the fear for robot-reviewers some have. Either because it gets monotonous and reviewer looses focus and starts making mistakes, or because of those who hunt a badge. I think the discussion should focus on new/better tools to fight that instead of denying raising the close vote limit and continue as it is.

If the main goal of a robot reviewer is a badge then maybe those who already have gold badge could get the close vote limit raised?

Today I did 40 reviews, its 6 hours before bed time so I could do some more. Actually I cheated. When I was at around review 38 I right-clicked the question link to open in a new tab and pressed skip. I did this x10! So I had 10 open posts from the review queue and I went thru them one by one. Not all needed to be closed.

Its a balance between opening a door for possible robots and reducing the review queue.

On top of many other suggestions here is a new one: I would agree with more votes to higher rep users and to those who already have the "wanted" gold badge, together with more audits and/or limits like one/some hour(s) forced review-pause after each 30~40 reviews. Another possibility is to have those excess votes above 40 not count toward any badges.

Can this ideas be tried and then evaluated if it was good or bad?

share|improve this question

closed as off-topic by gnat, Martijn Pieters, 3ventic, Emrakul, animuson Apr 29 at 23:05

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question pertains only to a specific site in the Stack Exchange Network. Questions on Meta Stack Exchange should pertain to our network or software that drives it as a whole, within the guidelines defined in the help center. You should ask this question on the meta site where your concern originated." – gnat, Martijn Pieters, 3ventic, Emrakul, animuson
If this question can be reworded to fit the rules in the help center, please edit the question.

8  
@apaul34208, People have been optimistic in the past. Some of the previous answers said "it will work" and pointed to the big fear of robot-reviewers, but the queue is growing. I think this need to be discussed again. I also add 2 ideas after reading the suggestions and fears of other questions/answers. –  Sergio Aug 31 '13 at 16:45
7  
If there is opposition to implementing Sergio's suggestions on a permanent basis, could we relax the restrictions long enough to reduce the backlog? –  HansUp Aug 31 '13 at 17:27
    
At the time those questions were asked the close vote queue didn't have entries just for close vote flags; it only accounted for posts that had 1-4 close votes. Once they moved close vote flags from the 10k review to the close vote review it will have both increased in the backlog, and the rate of increase. Before that had been done the queue had be steadily decreasing (although at an admittedly slow rate). –  Servy Aug 31 '13 at 19:54
    
Related: The Close Vote Review has been growing... –  Cole Johnson Sep 1 '13 at 5:44
4  
@ColeJohnson, yes... quite impressive that there were more than 30 posts on the subject and still many ideas (if any) have not been tried. –  Sergio Sep 1 '13 at 5:52
    
@Sergo because no one at "Stack Overflow and Friends" cares about it anymore. –  Cole Johnson Sep 1 '13 at 6:07
1  
Can you emphasise the last sentence (and explain that you're asking the current suggestions actually tried) to differentiate this from the duplicates (which you can't ignore; they're always on the front page). –  ben is uǝq backwards Sep 1 '13 at 8:20
    
@benisuǝqbackwards, sure. Just did that. Thank you. –  Sergio Sep 1 '13 at 8:25
2  
Another possibility which may have been mentioned above (I don't see it) - have those excess votes above 40 not count toward any badges. –  Adam Rackis Sep 1 '13 at 16:26
    
@AdamRackis, very good idea, added it also. Thank you. –  Sergio Sep 1 '13 at 16:29
6  
I lost the will at 2500 close reviews, mainly as it's too depressing seeing that number. –  hayd Sep 1 '13 at 18:59
1  
King Cannute lashing at the waves –  Gayot Fow Sep 3 '13 at 16:04
6  
interesting that out of 49 questions currently linked to this one, 1 is status-completed ("...Sorta"), 1 is status-declined and 47 (that's fourty seven) have no status at all. status-don-t-hold-your-breath –  gnat Oct 8 '13 at 7:45
    
the problem has been (at last) acknowledged and addresed by SE team: Enough fuzzying: let's let everything into the close queue and age out questions that don't reach a threshold –  gnat Apr 29 at 21:54

2 Answers 2

The Close Vote Review has been ever growing for as long as the review has been on the site. The problem isn't lack of awareness or lack of a long line of suggested solutions, the problem is a lack of implementation of suggested solutions.

Here are some of the suggested solutions:

Let me choose when I use my close votes
Let close voting from the review page not count towards your daily close voting cap
Close votes daily limit increase
Could we split up revision queues to get counts closer to zero?
getting things done: make the number posts to review more manageable for a single reviewer
Close votes policy review
How can the Close Vote review queue be improved?
Strongly separate duplicates from "the rest" in close votes review queue
Stack Overflow review tasks: close votes
Increase close votes in /review
How to make close-review queue less thankless
Should we raise the daily cap for close votes?
Close vote review queue reorganization proposal
Automatically close flagged questions
What can be done about the amount of close votes on SO?
What can be done about the massive Close Votes queue on Stack Overflow?
Ideas for the close queue?
Allow 1k users to review close votes, but count them only as 0.25 actual close votes
Reduce close vote expiration restrictions
Would raising the review limit help with the Close Votes queue?
Suggestion: close votes review queue
Automatically filter the Close Votes review queue to active tags for newbies
Daily Close-Vote review queue
Close Vote Review, additionally filter off-topic close reasons
Sort Order on Close Vote Queue
Daily close votes queue limit
Resort the Close Votes Review Queue
A proposal on how to influence the number of questions that are pending to be closed
Handling the close questions queue: weighted votes
New queue for moderators: questions that appear stuck in close votes review at SO
Proposed contest for close review queue posse
An alternative approach to the close queue
Give a broom to anyone who completes 10 close queue reviews successfully in a 24 hour period
Tweaking close requirements to manage the close queue
Fuzzy the number of questions in the close review queue, a dopamine for the shutterers
Get Community involved in the close process
How to handle ever-increasing number questions pending closure in the review section
A non-competitive, chat-based event for reviewing close-votes?
Quick-links on close vote review queue
Provide incentives for reviewing close votes in upcoming Winterbash 2013
Easy close review queue filtering interface
Another idea for close-vote queue rejuvenation: Show the number of items added today, not the total number
Show close votes on a question from the front page to help reduce the close vote queue
Show only the number of questions awaiting review from my filtered queue
For Reviewers: Notes and History
Reset the close vote queue
Reduce the number of reviews needed to complete an item in the CV queue
Display the number of pending review tasks created in the past 24 hours for the Stack Overflow close review queue
Allow direct linking to a filtered /review queue
Reward me for my community service
Reputation for close votes
Should we reduce the required reputation to review close votes?
Please streamline VTC-as dupe in the review queue

And here are some questions regarding the close vote review:

Huge close votes review queue on Stack Overflow
Is the close system working as expected?
How slow is the Close Votes queue being processed? And how large should it get daily when the backlog is gone?
Why is the close vote review queue so full?
Do we have a problem with follow-through on close votes?
Why are there so many close votes to review and so few of the others?
Stats of close votes review queue for SO?
Can we know the rates of adding and reviewing close votes?
From bull to bear - what can be done to decrease the mass of close votes?
Close Vote Funnel and Metrics
Can someone analyze the Close Vote queue on StackOverflow?
How can I make a bigger dent in close vote reviews?
Regarding the Stack Overflow close review queue
Close Votes review: I'm going on a strike!
Close Votes review: I'm NOT going on a strike!
How to tag questions related to size of The Queue?
Close Vote Review - Can we just get a Community Wiki on this?
SO Close Vote Reviewers (Next Meetup: Fri, Mar 7 @ 4:00pm EST / 21:00 UTC)
How many posts in SO CV queue aren't eligible for vote expiration, ie have no votes, only flags?

Original examples compiled by Gnat, borrowed from:
Strongly separate duplicates from "the rest" in close votes review queue

My point here is that the size and growth-rate of the close vote review is a painfully well known problem, with a multitude of suggested solutions, I think it may be time to finally address the issue.

Can this idea Can something be tried and then evaluate if it was good or bad?

share|improve this answer
13  
Or instead of any of that we could just close this question as a duplicate and pretend the problem doesn't exist –  Richard Tingle Aug 31 '13 at 17:23
8  
@RichardTingle I was tempted to do that, but I thought about it for a minute, and the impulse passed. –  apaul34208 Aug 31 '13 at 17:24
2  
@RichardTingle given the long list of questions on the subject, I doubted that anything new could really be said about it, but on the other hand nothing has really been done to correct the issue so perhaps it bears repeating. –  apaul34208 Aug 31 '13 at 17:30
4  
I sometimes wonder what if SE team simply have their reasons to keep status quo? "one may ask self a question, what would happen when (if) this queue becomes as short as other queues?" –  gnat Aug 31 '13 at 19:20
1  
@gnat I see your point... but isn't that what "On Hold" was for? why have a huge backlog on top of that? –  apaul34208 Aug 31 '13 at 20:32
3  
@gnat well put Sir –  apaul34208 Aug 31 '13 at 23:11
    
@apaul34208, should I add feature-request tag to my post? –  Sergio Sep 1 '13 at 10:48
13  
I don't have anything intelligent to say about this right now, but I wanted to drop by and mention that this post got seen and that I'll bring it up for discussion internally. (cc @Sergio) –  Anna Lear Sep 1 '13 at 16:47
7  
@AnnaLear, that's very good news. Many people have been asking for attention in this subject for a while. Thank you. –  Sergio Sep 1 '13 at 16:48
    
@Sergio any time you're asking that a feature be added or changed the feature-request tag would be appropriate –  apaul34208 Sep 3 '13 at 21:15
2  
Impressive list... many good ideas. –  Rikard Sep 5 '13 at 15:00
1  
I've posed notify users of possible reviews on toolbar as one approch to help people realize that there is something that they can do in the review queue. Just being aware of it may help. –  MichaelT Sep 12 '13 at 16:15
3  
@gnat Thanks for the edit/update. Perhaps its a good idea to keep an ongoing list here, till the issue is resolved. I'll try to keep an eye out for new ones as well. –  apaul34208 Oct 8 '13 at 16:07

Shog9, Tim Post, and I met earlier today to discuss the issues with the close queue and we all agree that we need to take some steps towards making it more manageable and, more importantly, more reliable and useful than it is today.

Unfortunately, there isn't just one thing that we could do that'd obviously solve all of our problems, and there are some additional challenges in balancing this between Stack Overflow and the rest of the sites in the network.

Before I get into anything else, I should note that the goal of any improvements shouldn't be to just get the queue size down. If we just want to see a low number there, there are easy "fixes" we can do - inject fewer posts into the queue, reset the queue and start from 0, etc. The queue size isn't the point in and of itself. Stack Overflow gets a LOT of close votes and "recommend closure" flags every day, so the size of the queue is likely to always be rather substantial. What is important is that the posts that enter the queue get reviewed in reasonable time and that we can see some progress being made.

It is certainly true that there are more posts entering the queue than there are leaving the queue due to completed reviews, especially since we redirected "recommend closure" flags from the moderator flag queue to the review system. Some posts are removed from the queue because close votes just age out, but that's not always a good thing. If we can't trust that the posts in the queue are getting enough (or any) eyeballs on them, we can't necessarily age the votes correctly either.

We have considered raising the number of reviews available to people, likely scaled with the number of their reviews that resulted in the community-approved action ("leave open" on questions that didn't get closed, etc.). However, we want more people doing fewer reviews a day to spread the work around rather than just overload the already highly engaged reviewers to ensure that nobody burns out and that as many people get involved as possible.

The review limit is still on the table and we haven't ruled it out, but we feel that we have to make some more substantial improvements as well. We have a few ideas, some of which may be unworkable:

  • more intelligent close vote aging that takes into account actions taken by users with close-voting privileges while they're looking at the questions with pending close votes or flags. (We are already working on this one. Expect a separate meta post once we've had a chance to vet and sanity-check the criteria we came up with against real data.)
  • more aggressive dismissal or aging of "recommend closure" flags
  • better advertising of the review feature to users who are able to vote to close.
  • we can possibly do better at showing people close reviews in tags they care the most about. Maybe add some "these posts need your help!" kind of message to the community bulletin on a per-tag basis.
  • split possible duplicates into their own queue with a better UI that'd make evaluating those flags easier. This doesn't so much fix the problem (beyond obviously lowering the number of items in the "main" queue), but it should make it easier to review those items.
  • we prioritize newer posts in the queue, so some of the older ones are not getting seen. We could vary things up a bit to try and get older posts reviewed.

There are many other suggestions. Hat tip to Gnat and apaul34208 for summarizing the meta posts on this. For the moment, we are moving forward with improving the vote aging criteria. Once that's in place, we'll re-evaluate where we are and pick the next area to focus on.

share|improve this answer
5  
Can we please get some additional filtering for off-topic questions? pretty please with unicorn dollars on top? –  apaul34208 Sep 4 '13 at 23:01
6  
"split possible duplicates into their own queue" -- I'd bet my 5,571 CV reviews against 1 that split would make a substantial difference –  gnat Sep 5 '13 at 8:51
2  
One quick point on the queue size; I wouldn't want the actual queue size to go down (you can hide some rather than age votes differently say) but the size of the queue and the ever increasing nature make it extremely psychologically discouraging to engage and review stuff. A "fake" reduction in queue size will re-engage people who've given up. –  ben is uǝq backwards Sep 5 '13 at 10:20
2  
a suggestion: resort the review queue based on the closed votes –  user221081 Sep 5 '13 at 10:20
1  
How about an increase in number of close votes based upon site reputation? That would follow a model similar to the delete votes where each 1k past 10k "earns" another delete vote. I think it would be fair to rep-cap the total number of close votes as well. –  GlenH7 Sep 17 '13 at 18:21
    
I understand the need to limit the number of daily reviews to avoid reviewers burn-out, but here is an issue it could be taken into account: today I reached my daily CV limit of 40, I'm not at work and fairly fresh, so I could have reviewed 40 more with no problem. Most probably there will be days where I won't be able to review any. Wouldn't a more flexible limit system help here?. Say, daily limit of 100, but week limit of 3000? (BTW, after reviewing those 40 I saw the queue count go UP by 100 after just a couple of hours - discouraging!). –  Lorenzo Donati Oct 8 '13 at 13:22
2  
@LorenzoDonati I agree that we should review the limits again. First, though, we're planning to try out a few other things, chief among them being attracting more people to do the reviews. We can't scale this very well by just having the group of people already reviewing doing more reviews. –  Anna Lear Oct 8 '13 at 14:20
    
Anna, was there an estimate for the impact made by flags to close? I ask because technically, these flags seem to be designed to push posts into queue "forever" (no expiration / dismissed only by voting) –  gnat Nov 14 '13 at 9:16
1  
Having a visual indicator when browsing a specific tag or set of tags that shows 3k+ users, "This vote has pending close votes" would also allow some of the newer votes to get acted on prior to the queue. Getting eligible voters to VTC prior to the queue would possibly increase participation. –  jmac Nov 14 '13 at 9:29
2  
@Anna regarding "better advertising of the review feature", making something in the main Stack Overflow page right banner might do the trick (where there are usually Career ads). –  Shadow Wizard Nov 20 '13 at 20:10

Not the answer you're looking for? Browse other questions tagged .