What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

Today, the size of the close-votes queue is 55k questions and growing. Is that a problem?

Which action could be taken to clean it up? It seems to be growing faster than Stack Overflow members are capable to handle it.

share|improve this question

closed as off-topic by gnat, Martijn Pieters, 3ventic, Emrakul, animuson Apr 29 at 23:04

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question pertains only to a specific site in the Stack Exchange Network. Questions on Meta Stack Exchange should pertain to our network or software that drives it as a whole, within the guidelines defined in the help center. You should ask this question on the meta site where your concern originated." – gnat, Martijn Pieters, 3ventic, Emrakul, animuson
If this question can be reworded to fit the rules in the help center, please edit the question.

65  
Reviewing them... –  Bhuvan Rikka 웃 Oct 18 '12 at 13:15
5  
Which action could be taken to clean it up? Reviewing them? –  SingerOfTheFall Oct 18 '12 at 13:15
21  
It used to be larger, Don't panic. –  Daniel Fischer Oct 18 '12 at 13:16
2  
@BhuvanRikka: I understand (and do reviewing myself), but it seems to be growing despite that effort :) –  Andrey Oct 18 '12 at 13:19
6  
@Andrey Umm... it's getting smaller, not larger. It might temporarily grow by a few dozen/score, but day-to-day it's been going down, not up. –  Andrew Barber Oct 18 '12 at 13:20
3  
It's down from at least 60k. So by this time next year it might be only 20k! But seriously, we're probably lucky it's going down at all, let alone the backlog. Because that is a massive backlog –  Ben Brocka Oct 18 '12 at 13:21
23  
If we didn't have to read what was being voted to closed, that would help –  random Oct 18 '12 at 13:42
10  
If anyone tried the review queue when it was first released and never went back because of the terrible load times, I want to ask you to try it again. Load times have dropped considerably in the past few weeks. –  Bill the Lizard Oct 18 '12 at 14:00
1  
@random nobody reads what was being voted to close –  gnat Oct 18 '12 at 14:18
10  
Lower the close threshold. –  j08691 Oct 18 '12 at 20:41
2  
It only takes four days for close votes to expire. If expiration was going to play a role in changing the queue length, it would've done so already. (Or it is doing so, and 55k is already the low value.) –  Pops Oct 18 '12 at 20:41
5  
Close votes expire at a rate of one every 4 days as long as the question has more than 100 views. Therefore, if everyone leaves for 2 weeks, it will be down to probably about 15k. EVERYONE TO HIATUS –  Wug Oct 18 '12 at 20:47
8  
@BenBrocka You still think it'll be 20k close votes by October? It's back at 57.7k and rising... –  Tobias Kienzler Jul 4 '13 at 12:19
8  
Well it hovers around 90k these days, and I don't know how much that will change given the apparently low limit on personal daily close votes * the number of people that regularly review close votes. –  Jason C Nov 6 '13 at 22:32
5  
@JasonC - it's not hovering around 90k, it's still steadily increasing, albeit at a slower rate than it was at one point. –  Spudley Nov 18 '13 at 9:16

25 Answers 25

up vote 161 down vote accepted

Which action could be taken to clean it up?

TL;DR Educate active reviewers to filter SO Close Votes queue by single close reason, with "duplicate" and "too localized" additionally filtered by their favorite tags.

Based on my experience, this leads to substantial productivity gain.

http://i.stack.imgur.com/0pyUG.png

Boring details on what led me to above conclusion are provided below; feel free to skip these.


Initially I've been going through close votes queue unfiltered. This worked just fine in all other queues I tried (dozen or two queues at SO and some smaller sites) and I saw no reasons to do it other way.

This queue felt a bit difficult from the beginning, but I thought I'll get used to it after a while. 300+ review actions passed, but it still felt cumbersome, at this moment I got curious why is that?

  • Discomfort I felt when running through queue reminded me of old review system and of the reason why I didn't want to use it: items in queue were all too different and I just couldn't get into flow because of that. This naturally led me to idea to give filters a try.

My testing of filtering by close reasons began with "off topic". Bingo! Review went smooth and fast, and most important, I felt comfortable - just about like I felt with all other queues in new system. I tried "not constructive", then "not a real question" - great, the magic is there.

Testing "too localized" though didn't go so well. It felt cumbersome again - just the same feeling of having to deal with too many distractions. This lead me to idea to try filtering it further. I added a tag I was comfortable with and the magic flow was back again.

Based on my experience with "too localized", I switched to testing "duplicates" filtered by favorite tag from the very beginning. This worked well, too. Well, to be precise, duplicates review went slower (much slower) than the rest: I had to study both questions, and in some cases I also had to check the answers to both questions. But (and this is very big BUT) what is important, it felt really comfortable - as opposed to my prior experience with unfiltered queue.

share|improve this answer
48  
Wow. I never even saw the "filter" link. Maybe besides educating users amongst ourselves, the filter itself should be more visible. For example, instead of the dropdown filter currently in place, maybe the "Review | Close Votes" page should have tabs that make it clear that one may choose what kind of close votes to support or deny, e.g. "duplicates | off topic | not constructive | not a real question | too localized | ____ [tag]". Okay, so my particular idea might be too messy, but I mean something to make the filter more obvious and attractive to use! –  Andrew Cheong Dec 11 '12 at 18:41
1  
I got to say it again: seeing just one question at a time, with a significant load time between questions, is a massive turn-off for me. I would gladly review a large number of questions if I didn't feel at every single step that it was going to be torture. –  Kerrek SB Oct 7 '13 at 22:46
    
@KerrekSB well in my experience of working with properly filtered queue this wasn't an issue. If memory serves 40 reviews were taking like 10-20 min to complete (can't tell more precisely since I dropped systematic reviewing about 4 months ago, after I decided that SE team just doesn't give a damn about queue size) –  gnat Oct 7 '13 at 22:59
1  
at 97 upvotes that came in through over this year, this answer seems to be steadily coming to +100. Guess that's going to be the first time when I would be unhappy to earn a Great Answer badge. "Can something be tried..." –  gnat Oct 22 '13 at 19:50
2  
There is no productivity gained by filtering when you hit your close vote limit in 10-15 minutes. –  Jason C Nov 8 '13 at 10:09
2  
@JasonC the point of this answer is to educate active reviewers, the more of them, the better, assuming that 200... 500... 1000 guys learning about this and using this knowledge, taking these 10-15 minutes daily, would drain the queue fast. This apparently doesn't work, partly because not many users read MSO (100 something guys who upvoted this answer is a drop in the ocean), but mostly I think because SE team takes no effort to make filtering easier to discover and learn about –  gnat Nov 8 '13 at 10:29
1  
OMG, it works! Thanks for teaching me about the existence of this feature. –  Ilmari Karonen Dec 28 '13 at 21:15
1  
I just adopted this method based on your answer and it works miracles for focus. –  Jan Doggen Jan 29 at 12:29
up vote 120 down vote
+100

To reduce the backlog:

define a function: close(p) = ....

where the right hand side includes the number of close votes, the total rep of all the close voters, the net vote score of the post, the number of answers to the post, and the age of the post.

In other words, 3 close votes from high-rep users, on a negative-score question, with no answers, should just close the thing. Or some rule like that. Then we could concentrate on content that had some chance of having value.

share|improve this answer
48  
Does seem like a good idea. Perhaps this sort of close vote weighting only occurs after a certain time has passed? That way we could try to reach the vote threshold naturally, and if we can't, then trigger "overtime" gameplay where the close votes are weighted by reputation? –  Jeff Atwood Oct 18 '12 at 16:32
3  
@JeffAtwood I did list 'age' in the list, even if I failed to mention it in the example. –  Rosinante Oct 18 '12 at 22:36
7  
Instead of total rep of all 3, I'd suggest looking at the "user tier" that they're in (i.e., 10k user, 20k/trusted user, etc.). We don't want 1 vote from Jon Skeet and 2 from other 3k users closing it, do we ;) –  Lorem Ipsum Oct 19 '12 at 19:46
9  
@LoremIpsum what? yes we do. –  AAA Apr 25 '13 at 17:38
2  
An important question to ask is how many questions in the queue this would actually impact. Depending on the actual function, it may not be that many. –  Nathaniel Ford Jun 10 '13 at 19:37
    
This question was just bountied, and I realized I had already upvoted this answer at some point in the past. I believe I was going to ask a question and then found this same sentiment so just upvoted. I really like this suggestion. However, when to apply the metric seems to be in the air a bit. Jeff suggests a certain amount of time passing and I think that will work. How about when the close votes start to fall off? –  Travis J Oct 21 '13 at 18:16
    
For example, if there are 3 close votes and one is going to expire time-wise, then perhaps the weight of the 3 votes could be assessed based on the voters' reputation and perhaps extra weight (+1) could be given for "trusted" users (>20k). –  Travis J Oct 21 '13 at 18:17
4  
I wouldn't use the user tier or the rep. I hate that the q/a based rep system is tied to site privileges. The source of rep is completely unrelated to the privileges the site judges you should have based on that rep. Instead, perhaps a new concept of e.g. "moderator rep", increased when, say, you make helpful flags, or the community approves or of your votes. Things actually related to moderating the site. –  Jason C Nov 8 '13 at 10:11
    
@JasonC Hmm, on the other hand most will probably be off-topic votes. The persons that answer questions should be able to differentiate between on- and off-topic for specific tags. –  Maarten Bodewes - owlstead Aug 6 at 12:28

Take a minute here for a sanity check:

close review queue stats

Update as of 10/24/2013: 86k need review, 1,151k all-time reviews

There's a massive amount of reviewing being done there. It takes 5 voters to close or 3 "do not close" reviewers to kick a question out of the queue without closing. And yet, folks are making some impressive progress.

Frankly, I don't particularly think this needs to be sped up any - closing is hard by-design. If it takes a while to get through the backlog, that's probably ok...

Update: review task creation / deletion over time

In response to LittleBobbyTables' concern, here are some graphs that may help to illustrate the problem with clearing this queue:

Close review tasks created

Close Review Tasks created

Close review tasks deleted (either completed in review, closed or deleted elsewhere)

Close Review Tasks deleted

Review tasks are created in response to close votes or flags - these have been going up.

share|improve this answer
11  
I started about 3 - 4 days ago to help with this review process. So I have a small suggestion: how about increasing the maximum allowed reviews / member / day from 40 to something more ? Maybe 80 or 100 ? Just temporarily until that huge queue gets smaller... much smaller. It would go faster. All the upsides with none of the downsides :) . (PS: I didn't know where to suggest this, so I wrote it here) –  Radu Murzea Jan 20 '13 at 20:33
2  
I dunno, last month we were at 46.6K~47.2K, and now we're over 50K again. My problem is that I see so many bad questions during the day, I tend to run out of close votes before I can get to the review queue. –  LittleBobbyTables Mar 7 '13 at 16:38
    
This might be a problem, @LittleBobbyTables - see my edit. We'll have to keep an eye on it. –  Shogging through the snow Mar 7 '13 at 17:53
    
that recent spike in reviews created, could it be contributed / correlated to Close all the typo questions activities? –  gnat Mar 7 '13 at 18:58
    
The timing lines up pretty well, @gnat –  Shogging through the snow Mar 7 '13 at 19:14
15  
Downvote because "And yet, folks are making some impressive progress." is plain false. Maybe 'getting a lot done,' but progress against the review queue size? Nope. –  AAA Mar 31 '13 at 5:26
3  
Status check, June 2013: 54.5k items in queue. Progress is still negative. Queue is still growing. –  AAA Jun 12 '13 at 0:36
2  
Now the queue has gone up by another ~10,000 and shows no sign of going down, does this need to be reconsidered? –  Matt Aug 2 '13 at 11:28
2  
Yes, @Matt –  Shogging through the snow Aug 3 '13 at 17:08
1  
recently, I am pondering over an idea that maybe, SE team has a bunch of good reasons to keep large CV queue backlog and, OTOH, no good reasons to drain the queue (yet). I wouldn't mind to see it proven wrong but so far, this is not what I observe –  gnat Aug 7 '13 at 13:07
3  
Today it's 83K. Just saying. Maybe it's time to think about splitting up the work in a different way. –  Aaron Bertrand Oct 9 '13 at 14:20
    
2 months from this post and its up to 116k, it's growing exponentially. Wouldn't a 40 to 50 limit change be reasonable? –  scrowler Dec 14 '13 at 10:45

At the moment we can filter by tag and by close reason, but off-topic has 7 sub-close-reasons.

It may improve the workflow a bit to add additional filtering and be able to narrow it down to one or two sub-close-reasons.

enter image description here

Related: Close Vote Review, additionally filter off-topic close reasons

share|improve this answer

Just as an input to this discussion, a small progress report.

After a couple of weeks (or maybe more? don't have the energy to check) on Stack Overflow spent maxing out my close vote queue reviews filtered by the cocoa and objective-c tag, today I got the very nice message

There are no items for you to review, matching the filter "[cocoa] [objective-c]"

(thanks to @gnat above for attracting my attention to the filter)

A couple of random observations from this marathon clicking exercise, in no particular order:

  • It is generally recognised that bad questions are a problem, but bad close votes are nearly worse. They contaminate the close vote queue and take up at least 4 persons' time to read and analyse (especially in the case of duplicates). A mediocre question with no votes and no answers will just sink down to the bottom of the "sea" of questions, while the same forgettable question with a random, half-warranted close vote now (thanks to the review queue) requires 4 or more people's time and effort.

  • There is a pattern of questions that are regularly closed, for instance requests for tutorials. I wonder if there is a possibility to catch such questions (based on an automatic filter) and ask the poster to verify that the question is not simply a question of that type. Naturally, if they tick a box, the question will be posted in the normal fashion.

  • The review queue definitely does push one into piling votes on top of the first one. After having cast several hundred votes I am sure that has happened to me, too. The point is, I am not even sure it is a bad thing; if you trust the user who cast the first vote and did the work of analysing the question and, in the case of a dupe, searching for the "best" dupe to point to, that is a great help. Maybe it should be harder to cast the first vote (open the close process) and/or easier to participate in the subsequent votes?

  • There is a lot of junk out there, and I have to say that I don't subscribe to the idea that if just a single person out there is helped by a question, it is worth preserving. The effort to curate those low-quality, localised questions just doesn't warrant it.

  • I have moaned in the past that I thought it was too much to ask the community to wade through more than 50.000 questions in the review queue for no real reason (what is the point in putting on hold a 9 months old question that no one has looked at since the day it got posted?) Now that I have actually done it (for the tags mentioned) I can safely say that it is not just something I think. I am certain: it is largely a waste of time and effort. It does have a certain therapeutic effect, though, a bit like that of a punching ball (you get to close a lot of stupid questions, even if you don't have the option to call them that), but I don't think that was the intention with the queue.

share|improve this answer

After all this time, I wanted to add a view of why people don't visit the queue -- or -- at least, why i don't visit the queue.

I have a limited number of close votes. I visit the queue, I use them all, and I feel like I'm the bellhop in Hilbert's Hotel. The amount of work to do remains infinite. I have zero feeling of accomplishment.

In my view, the size of the queue is a view into the ongoing torrent of crap dumped onto the site. I'm willing to keep trying to help deal with it, but I need a larger gun if you want me to help shoot this many zombies.

share|improve this answer

My suggestion: Encourage people who create close votes to also do some reviewing.

To clarify:

When someone votes to close a question, I suggest that the system should check their reviewing records; if they haven't reviewed any close votes recently, then a polite notice should be shown asking if they would consider doing some reviews to shorten the backlog as well as just adding to it.

I would hope that this would encourage more people to do a few reviews who wouldn't do so otherwise.

(the above assumes that the user has permissions to do reviews, of course)

[EDIT]

I see this working a bit like the voting system does now. Right now, if you mainly vote on answers, you might occasionally get a message when you vote on one saying "You haven't voted on a question lately. They also need some love." (or words to that effect).

So how about doing the same with close votes -- if someone's issued a number of close votes without using the review system, they could get a gentle reminder like "You haven't done any reviewing lately. Please help reduce the workload by reviewing some questions with close votes."

It's not intrusive, it only pops up once in a while, and it doesn't force anyone to do anything, but it might just help do something about the backlog.

(by the way, the close vote review queue is growing rapidly at the moment; it's pushing toward 70k. It's gotta be seen as a problem sooner or later, hasn't it?)

share|improve this answer
    
I'd love to –  Tobias Kienzler Jul 12 '13 at 13:17
    
You do know that lots of people hate that notification provided by the voting system, right? I doubt it'll be all that motivating. –  Cody Gray Aug 10 '13 at 12:20
2  
@CodyGray: if its hated, it's because of how often you see these things. I did say it should be un-intrusive; I'm not proposing it would appear often. If you see it at all, it would mean you're already engaged with moderating because you've made a bunch of close votes (probably quite a lot), but haven't touched the reviews. And even for those people, I wouldn't expect it to show up more than a couple of times. You only need to engage a relatively small number of people for it to be effective, so it doesn't have to be popping up all the time. Definitely a lot less than the current voting popup. –  Spudley Aug 10 '13 at 12:30

Let's take a look at the problem. As of today, this queue is up to 81.8k questions in the queue. Assuming each has 1 closed vote, that means that a total of 320K actions are required to close all of the questions. Given 20 per day, that means that 160K people would have to clear out 20 in order to close the queue. Then after it is closed, then more people need to review them than have been doing so. So, how can these get reviewed? There are a few ways:

  1. Increase the number of people reviewing.
  2. Decrease the amount of things that need to be reviewed.
  3. Increase the number of things that can be reviewed per day per person.
  4. Decrease the time per review required

Okay, given these options, how could any of them be done?

Increase number of reviewers?

Close vote reviewing is a painful, thankless task. Sure, it's the easiest badge of the various custodial badges to earn, but a closed vote decision can be quite difficult sometimes, depending on the nature of the question. Perhaps there should be some incentive to encourage more reviewers, either in the form of reputation, or increased privileges.

Decrease the number of things to review

There are a couple of ideas that I have that fit into this category. Either somehow things need to be automated better, or the number of close votes could be reduced. As the latter could be dangerous, perhaps we should just allow certain users to have more closed votes, perhaps those with a very high reputation, or maybe those who have reviewed more closed votes overall. Both help to filter out the relatively new closed voters, and leave those with more knowledge of how the site goes more power to help it out.

Increase the number of things that can be reviewed

This improves the situation so long as you don't burn people out. As the pile is getting deeper, it could be a quick short term solution.

Decrease time per review

Filtering by tag or closed reasons, better displays to help users figure out what's going on, anything to make the task of reviewing easier increases the likelihood that a user will use all of their closed votes per day, and come back tomorrow and use them too.

I'm sure there are other ideas out there, but here's my preferred method:

  1. Allow users with a Reviewer Badge to get 2 close votes, and with a Steward to get 3 closed votes (Might need to be tweaked, but I'm putting out something here)
  2. Increase the number of reviews for the closed queue, at least for a while.
  3. Do some filtering automatically. The top questions I look at should be for tags I am familiar with. This will make the review queue simpler to manage, as I won't have to learn what the question is about before I look into it. I know that I can filter them manually if I want to, but it's not that obvious to a new reviewer, and thus I think it should be done automatically.
share|improve this answer
3  
I think your general suggestions are good - but I disagree with the premise that a suggested edit is easier to handle than a close vote. My experience is that suggested edits are reviewed much more haphazardly, as it is much subject to personal opinion with regards to ie. what is to minor an edit and what is an attempt to comment on a post. A lot of the bad posts are much more binary, IMO, with this as just one of many, many examples. –  user213634 Sep 26 '13 at 17:45
    
@AndersUP: Fair point, I suppose each person has their own preference as to which is easier. I should point out that the suggested edits queue doesn't have this problem, either due to fewer questions being put in to the queue, or they just seem to get reviewed easier than closed questions queue does. –  PearsonArtPhoto Sep 26 '13 at 18:10
    
I think fewer questions are being put in that queue. But generally, I don't think I've ever seen any of the other queues having 100+ items in them. –  user213634 Sep 26 '13 at 18:27
    
@AndersUP: That's largely because the close questions queue gets piled on faster than it can be removed. But I have seen at least the Low Quality queue have more questions once, when it was first created it would dump a batch around UTC 0000 there. –  PearsonArtPhoto Sep 26 '13 at 19:00

Another possible solution might be to bring questions that have more votes to the front of the queue. In other words, if you have 4 close or 2 leave open votes on a question, if it gets brought to the front of the queue, it is more likely to be resolved quickly.

share|improve this answer
3  
on the other hand, questions that have more votes are more likely to be closed without a review job. –  Jan Dvorak Jul 3 '13 at 12:24

Disclaimers: Though I am relatively new as a registered user of the site, I have been using it for a long time and have read a lot of posts that have been closed for various reasons. Also, since I can't yet participate in close-queue reviewing, I certainly can't speak of how easy or hard it is to do so.

Boring argumentation

Most propositions made here discuss how to deal with the close-queue as it is now. But it is even important to reduce the number of questions that do enter the queue in order to eventually reach the end and then keep up without tremendous efforts.

Now, a quick look at recent questions that were voted to close shows that a large number of these questions come from users with no reputation or almost none. A person learning a new language or technology, or even learning programming for the very first time, either at school or by following whatever tutorial found somewhere, is certainly the archetype of these users.

I don't think it is pertinent to discuss if these persons are being too lazy. It is obvious, to us at least, that the answer they need is lying somewhere around. But when you know very little, it might be hard to decode answers that are not tied exactly to your situation. So they come here, expecting to receive help. They are not considering the fact that their questions should provide common, persistent knowledge for others. And even if they would consider that aspect, their view of "knowledge worth sharing" is certainly tainted by the fact that they indeed have a very limited knowledge of the technology being learnt. Is it bad that they come asking for help? I don't think so. But their questions and the associated answers are certainly not assets to the Stack Overflow community.

So how does all that might help the present discussion? It basically tells us where to put efforts in order to reduce the influx of to-be-closed questions. These persons are looking for answers cut precisely to their needs, which they won't find anywhere. Stack Overflow is very popular and very well known for offering quick answers to questions being asked. Add those two facts, and you realize that you can't expect these users to stop coming here, nor can you expect them to stop asking low-quality questions. Not providing answers to these questions will not discourage students from coming here. But having a few persons trying to provide a quality answer (the kind that guide the user toward better understanding, rather than the kind that is to be copy-pasted), while five persons are reviewing the very same question to vote a close is certainly a waste of everyone's time.

So here are my suggestions:

  1. Have users posting questions flag by themselves if their question is of the "homework-or-of-little-interest-to-the-rest-of-the-community" type. Flagging as "homework" should not prevent posting a question (otherwise, people will simply repost without flagging, so we would be back at square number one). This question should be very prominent for first time posters, then could be replaced by a subtle checkbox once users have posted a few questions that has received community support (let's say vote up on two questions, or valid answers on at least five not-closed questions).

  2. Define a review-before-show-up-queue, where questions from low reputation users are sent before being officially displayed. Accepting a question from that queue would require much less votes (let's say two, at most) than closing an existing post, and thus, require much less work from the community. Acceptance of posts should not be based on the fact that the question is or is not a duplicate, but rather that the post is correctly flagged as "homework" if it appears to be so, that it actually is a question, that the question demonstrates some effort in understanding the problem (that is, the question does not appears to be a copy-paste from the problem statement), that the pertinent material, and only the pertinent part, is provided (existing code is there, but not a whole project), etc. Actually, all these expectations should be displayed upfront to first-time posters, with checkmarks on each, so that they can provide their questions in the correct format upfront.

  3. Add a special icon on "homework-style" posts in all questions lists. Or allow users to hide them. As a last resort, it could even be considered to move these questions to a distinct site. Something along the line of "programation-coaching-exchange". Anyhow, users that spend time answering questions should decide for themselves if they do want to spend time coaching beginners, and guests coming to the site should view higher quality questions, not beginners questions, so they think of Stack Overflow as a place for quality knowledge, not for Computer 101 course complement.

  4. Add a reminder on questions flagged as "homework-style" indicating that "cut-and-paste-ready" answers are not welcome. These doesn't help anyone. Coaching is the correct way to go.

  5. Automatically close "homework-style" questions after two weeks of inactivity. They could even be deleted after, let's say, 4 months. That is well enough for people to refer back to the answer they had to a question asked earlier in a trimester. Or for teachers to look around for activity that could be interpreted as cheating.

  6. Encourage "homework-style" posters to go around and reply to some other "homework-style" questions. Offer badge for "cooperation". That will reduce the charge of these questions on the rest of the community, help them build stronger knowledge, and help them view the other side of the community aspect of Stack Overflow.

Well, that's it for now. Hope that helps.

share|improve this answer
3  
While I don't get this whole "homework style" mechanism, I think that diagnose is correct. Also, the review-before-shown queue seems a good idea for questions from first-time users, and definitely something has to be modified about the question input interface. After discussion with @Servy under my answer I'm currently gathering data from this year to check whether indeed such questions are mainly responsible for the close-queue flood. –  BartoszKP Oct 24 '13 at 20:34
1  
Well, I assume that most people asking questions related to homeworks or tutorials would not lie on that fact if the question is asked directly. People already write things like "Ok, this is my first program, please be patient..." So having 5 persons review a post only to conclude something that the poster would have said very honestly if asked so seems a bad approach. And then, once we know that a question is related to a homework or some beginers tutorial, well we might just as well offer answers that better suit this fact, by acting as teachers, rather than coworkers. –  jwatkins Oct 25 '13 at 14:59
1  
But the quality of a question doesn't depend on whether it's a homework or not. It's the content that matters. –  BartoszKP Oct 25 '13 at 15:01
    
That's absolute right. But I'm pretty certain though that there is some corolation between "homework-style" and "low interest". Some questions asked in the context of homeworks might indeed be of high quality and of high interest as community's common knowledge. But I'm pretty certain (may be someone might attempt at demonstrating this fact) that most homework style questions are somewhat redundant with existing questions, presented in a way that is way too much specific, and can (somewhat mistakenly) be interepreted as lacking research from the poster. –  jwatkins Oct 25 '13 at 15:08
    
And then, I might add that when good "homework" questions occurs, they wight still be up-voted by anyone with that privilege, and then, they be automatically moved to the main questions area. The objective here is certainly not to disallow beginers from asking questions or to obtain answers. But I believe that these questions account for a significant portion of the "no question vote", "voted-to-close" and "no solution ever accepted" pools, all of which appears to be way too lage. And, let be honnest, qualified people tend to be less inclined to spend time offering quality answers to them. –  jwatkins Oct 25 '13 at 15:16
3  
All right, so I guess what I don't like is only the name "homework question", as I feel it represents a more general group of questions. But to be fair I don't have a better name for this other than "I don't know what this site is about, and I probably don't know what I'm asking"-questions or "I'm not good with google"-questions or "I don't have enough knowledge to ask google the right way"-questions. :) Anyway, I like the hierarchical queues idea. –  BartoszKP Oct 25 '13 at 15:41
    
This answer have given me an idea which I'll try to formulate into a concrete feature request once I've got enough statistics from SE database. I think hierarchical queues is the way to go, and so the bounty goes here. Thanks! –  BartoszKP Oct 27 '13 at 18:00

... or, you know, they could just give us more close votes.

It's 9 AM here, and I'm already out of close votes for the day. On most days, fifty votes just isn't enough to deal with new questions plus questions in the Close Vote Queue.

I don't need a prize, or a badge, or even thanks... I just want more freaking votes.

share|improve this answer
2  
Good point, but a duplicate :) –  BartoszKP Oct 24 '13 at 13:52

Automatically prioritize posts in the review queue that are tagged with either the users favorite tags or tags selected by the user. This could be limited to e.g. 3 tags if performance is a problem.

If there a no more posts with the specified tags, show other posts (without having to remove the "filter", which is just an annoying UX).

Similar suggestion, Feature Request Meta Post

share|improve this answer

I get the impression that except perhaps for test questions we're mainly getting recent questions to review. I suggest that once you've handled the current maximum 40 questions in a day (or perhaps earlier) you start processing the from the back of the queue, or just generally give us more old questions to process, and probably a higher limit per day in response.

share|improve this answer
2  
Old questions that no one's looking at aren't really hurting anyone; the new active ones are the important ones to close –  Richard Tingle Nov 15 '13 at 9:18

I have another suggestion to make:

Let vote-to-close flags count as silent close votes and convert them to real ones once they have been validated by a user actually voting to close.

Once a VTC-flag gets raised on a question the question enters the review queue. Now the post needs five votes by five different reviewers to be actually closed. Lets say a question enters the queue with two flags raised. A user reviews the question and votes to close. After this voting the close vote count of the question would actually be three, because the two flags cast have been converted to actual votes.

Consequently spoken: When a question has an active close vote (one that has not aged away) a new VTC-flag cast on that question should automaticalle be converted to a close vote and be validated.

To make a long story short: Once a user with the necessary privileges has decided to cast a close vote on a question, let others (whether they have the closing privilege or not) jump on the train by flagging the post.

share|improve this answer

Allow a metric other than filtering by tag. Such as searching with a text box or sorting by date or sorting by OP reputation.

It doesn't make sense to constantly be reviewing content that was posted 49 minutes ago and is still being edited.

share|improve this answer

Copied from Reduce the number of reviews needed to complete an item in the CV queue


There are currently 95.3k items in the Close Vote queue, and 2 items in the Reopen Votes queue.

The number of reviews needed to close a question from the CV queue is 5, which is ridiculously high. Here's a stat:

In the past 90 days, 46,506 close flags have been processed; 45,707 were marked as "helpful" (resulted in at least one close vote) and 34,000 have resulted in questions being closed.

That's 73% accuracy from <3k users flagging; i.e. the people we believe to be least-able to recognise a close-worthly question are already getting 3/4 of them right... do we really need a further 5 >3k users to back them up?


Here's my actual :

Let's find a number n, such that requiring only n reviews to end a review results in the same action being taken as requiring 5 reviews, y% of the time.

If we require only 3 reviews instead of 5, and we end up with the same outcome 95/90% of the time as we would do using 5 reviewers, why aren't we doing that already? Yes, I just made those numbers up. Shog help me..


For those that are dubious as to whether this would open the floor for questions wrongly getting closed, let me calm you down with this;

  • Firstly, don't forget that we have the ability to "Vote not to Close" in the review queue. In the normal question view we don't have this. This is why the bar to close can be lower inside the queue than it is out of it, because people can vote for both sides.

  • Secondly, my feature-request is not too different to how we already handle the Suggested Edits queue, and a wrong suggested edit is just as reversible as a incorrectly closed question. Hell, we even have a re-open queue; and it's empty. Let's use it.

If you're still struggling, then how about I knock you clean out by suggesting we add the ability to flag a question for reopening. This would force the question into the reopen queue, and handle the small percentage of questions that incorrectly get closed. It would also allieviate a proportion of the flags moderators have to deal with at the moment. If we're worried about the question-owner spamming the flagging option, only enable it for non-owners.

share|improve this answer

I have some suggestions and ask for more here:

  1. Review queues only contains the post which have already two close votes (and asked in the last seven days, but older post should be considered with one vote), as one close vote can not always be correct (some users cast vote just because they do not get the post).
  2. Privileges to close the post directly should be at a level where more users can avail it. (Obviously at the level where community can trust them).
  3. Three votes of users with more than 5000 reputation points should be enough to close the post.
  4. A pre-filter apply to the review queue for the users on the basis of tags they are most active in (If I am active user of jQuery give me list of it, so I don't need skip).
  5. Increase the threshold to review the queue from 40 to 60 (or somewhat more suitable)
  6. Highlight the close link for more visibility so more users can cast close vote directly from a post.
  7. Anti-close votes like Leave Open should be on the post itself, gets visible if a close vote is cast, so a wrong misleading vote can be ignored directly from the post without being reviewed in a queue.
  8. One reputation point over 10 reviews (or 10 points over 100 review) will motivate more users to review.
share|improve this answer

Here are my thoughts (posted under another similar question):

  1. In the close queue, show questions based on the reviewer's expertise. (Yes we have the manual filter but it has limitations, such as only allowing 3 tags max.)
  2. Make the close queue more transparent.
share|improve this answer

And some more suggestions:

  • Allow more people to review close votes. 18k users have > 3k reputation (Nov 2013), 33k users have reputation score between 1k and 3k. I would suggest to allow users with 1k reputation on SO and 10k global reputation to review the close votes.

  • Allow people who already have 1k close vote reviews (and have earned the gold badge) to review more questions per day. These users can't earn another badge so they wouldn't try to cheat the review system.

share|improve this answer
2  
-1 for first para, +1 for second - you need this a separate answers –  Mark Nov 7 '13 at 10:02
    
@Mark I have seen that other users has give several suggestion per post and thus I wrote my suggestion in one post –  Dirty-flow Nov 7 '13 at 10:18

A suggestion:

Age all posts out of the Close queue after two weeks. The rationale is that if five close votes cannot be accumulated against a post in two weeks, closing the post is likely a dubious decision anyways. Additionally, the close votes against those posts would also be cleared, enabling users if necessary to attempt a reclose of the question, without having to wait for an edit.

Optionally: As a corollary, one might argue that the same criteria be applied to the re-open queue: age out all posts two weeks after an initial re-open vote. However this queue seems to be unmanageable, so symmetry may be unnecessary.

This would not interfere in any way with the ability of the community to rapidly close (and if necessary delete) posts that absolutely must be dealt with quickly. In fact, this proposal might assist in having those posts stand out, enabling them to be dealt with more efficiently.

The immediate effect would be to clear a large number of posts from the Close queue, and with the aging mechanism in place, hopefully allowing it to be maintained at a reasonable size. As mentioned by others, the advantage of seeing a manageable Close queue is increased community interest in maintaining it at a manageable size, presumably allowing more poor quality posts to be closed, more promptly.

If this were to be undertaken on a trial basis, what is the worst that would happen? 74,000 or so pots (less those nominated for closing within the past two weeks) would be cleared. Most of those posts weren't going to be reviewed in less than a few months anyways.

If this change is adopted and becomes successful successful, one could imagine creating a separate long term Disposal-Review queue with the aim of slowly working through and assessing old posts. If they were sorted inversely on a quality measurement, the likely worst offenders (or at least, best candidates for action) could be dealt with soonest.

update: The underlying assumption behind this proposal is that the Close Queue is broken, and needs fixing; that a mechanism which worked well for several years has now been overwhelmed by the site's popularity, and needs refreshing.

share|improve this answer
2  
Close votes already age meta.stackexchange.com/questions/136989/… Some tags you have to wait because of low traffic. That shouldn't make them immune to close just because the close queue is a large number –  random Sep 2 '13 at 23:07
    
Thank you, I wasn't sure if that happened. However, I am proposing aging the queue itself, not just the close votes. I was emphasizing that the close votes themselves should of course age out at the same time, –  Pieter Geerkens Sep 2 '13 at 23:10
2  
I was about to suggest something very similar: set all close votes to expire, regardless of the number of views. Either that, or convince us all that we shouldn't be frustrated by the size of the backlog... @random Currently, low-view questions may not be immune to closure, but being left buried in such a huge queue certainly doesn't help. IMO the close vote system works better for active questions, perhaps we should embrace that. –  bfavaretto Sep 3 '13 at 2:46

From my answer here:

I think a contest is the right approach, but it needs to be done in a big way.

First, set an amount of time, I was thinking two months, though 45 days is fine. If you do that starting next month, you can end things at the end of the year, but that's not crucial.

Second, you give out one-time permanent badges for the contest commemorating their achievement forever. Let's say for 250, 500 and 1000 closes in that timeframe (obviously you should probably remove the review limit for the duration of the contest).

Third, you give out either rep or even specialer one-time ranked permanent badges for the top 3 closers.

Make it as big as the Winter Bash Hat thing, go wild.

share|improve this answer

Would it not be possible to change the minimum amount of required rep to 2k instead of 3k? I think 2k is already quite hard to get, and proves that you know your way around SO and should be capable of deciding if a question needs to be closed.

More reviewers obviously helps trim down the queue.

share|improve this answer
    
No, it is not hard to get. –  Johannes Kuhn Nov 15 '13 at 8:54
1  
@JohannesKuhn it certainly is for me. It even took me 13 days to simply get 50 rep when I just started out here. After 4 months of membership I now have almost 600 rep, and gave 63 answers. –  Dark Ashelin Nov 15 '13 at 9:10

The problem still persists - currently the queue has 85,200 questions in it. I like this answer by PearsonArtPhoto. I would add another suggestion to it:

Should we make the minimum reputation level higher for asking questions? This would significantly lower the amount of bad questions that are flooding the queue.

Most of the bad questions are not a result of users being stupid, just lazy. If they would be forced to participate in the community for some time, before being allowed to ask a question it would result in two things:

  1. They would ask a question only if they really need an answer. If the amount of time needed to earn, let's say 5 or 10 (or 2 or 20 - just examples) reputation was worth it for the user, then probably this question is not trivial.

  2. They would learn more about this community just by participating it, not by reading the help center (let's say this clearly - almost NO ONE reads the FAQ before posting their first question).

Update:

I've made a little research to find out how many "good" questions would be lost, if there was a bar. The procedure was as follows:

1) I've queried SO database to retrieve all closed questions since 2013-01-01 till 2013-10-20 and all opened and up-voted (with score at least 1) for the same time range

2) I've filtered the closed questions to get "questions from fresh users". A "question from fresh user" is a question with creation date distant no more than 5 days from user's account creation date.

3) I've accumulated all user IDs from the previous step, to produce a list of "users who asked a question within 5 days from their account creation which was closed"

4) I counted the "good" questions created by these users. "Good question" is a question that is not closed, and has score at least 1.

The results are:

1) There were 95007 closed questions and 600270 opened, up-voted ("good") questions.

2) 22514 of these questions were "fresh" (23%).

3) These "fresh" questions were asked by 21016 users.

4) These 21016 users asked also 13652 "good" questions. Overall number of "good" and "fresh" questions is 90092.

Out of 600270 "good" questions 13652 is around 2.3%, and total percentage of "good" questions from "fresh" users is 15%. Therefore, it seems if for given time range a bar was introduced probably(*) there would be 23% less questions to close, and 15% good questions lost.

If we define "good" question as a question with score 3 or more, then only 2.1% would probably (*) be lost (12637 all opened questions with score at least 3).

If it would be possible to filter out users who asked a "bad" question not using a global rep bar (for example: introduce rep bar after first asked question was downvoted heavily or closed) than only 0.3% of "good" questions would've been lost (1986 questions with score at least 3 from users with closed "fresh" questions).

(*) it's "probably" because it is still possible that these questions would have been asked anyway, after the user would reach the bar

Explanations:

  • why the time range?

    I started downloading the data on 2013-10-20, and it took me a while.

  • why check day-distance between asked question and account creation time, and not reputation itself?

    It is practically impossible to retrieve user's reputation value for a distant past.

  • What is the query for closed questions?

http://data.stackexchange.com/stackoverflow/query/152649/list-of-closed-quesions-from-2013

(note the throttling - you may need to run the query more than once, uncommenting the id condition with appropriate id each time)

  • What is the query for opened up-voted questions?

http://data.stackexchange.com/stackoverflow/query/143859/open-upvoted-questions-2013

(note the throttling - you may need to run the query more than once, uncommenting the id condition with appropriate id each time)

share|improve this answer
7  
1) This would reduce a lot of bad questions, but it would also get rid of a lot of good questions. Many really great questions come from low rep users. Next, this will also just result in a flood of really bad/duplicate answers of people rep farming (way more than we see now) to get enough to ask questions. –  Servy Oct 21 '13 at 16:24
1  
@Servy You seem to spot a hole in every idea. Even if you're right, this is not very constructive and doesn't solve the problem :) Ad rem: I'd argue against your point - if the questions are good, they will be asked eventually. Probably yes, some users will be discouraged, but the lack of good questions is not a problem. The problem is the flood of bad questions. –  BartoszKP Oct 21 '13 at 16:29
4  
If you're unwilling to have users discuss the problems or drawbacks of your ideas then you shouldn't be suggesting them. By doing so you're asking others to discuss their potential drawbacks. It is in fact very constructive; to simply ignore the drawbacks or problems in a proposal results in implementing features that cause significant problems. With a bar that high many people just wouldn't bother to ask questions, and with so many fewer questions coming in it would be much harder to earn reputation. The whole system would just break down at that point. –  Servy Oct 21 '13 at 16:33
    
@Servy All right, my point wasn't that discussing is bad. It just that in this particular context discussion lasts for one year or so and seems like no attempt was made (?) to at least try to solve the issue. It's great that you can see possible flaws in my proposal - but is there anything better you have to offer to actually solve the problem? "With a bar that high" - how high? I'm not proposing any particular number, these were just examples. –  BartoszKP Oct 21 '13 at 17:44
    
Also: There are other ideas here which are great (not only by my judgement, but also looking at the votes) and nothing has been done. Hence my idea to bring the attention to this topic again :) –  BartoszKP Oct 21 '13 at 17:44
    
I never said that nothing should be done, or that all of the proposed ideas were bad. I'm just saying that this proposal is very bad, and nothing more. I'm not obligated to have a better proposal to comment on why this one would be bad. Doing nothing would be better than implementing this proposal. You specifically mentioned a bar of 100-500 rep. Even a bar of 50 or less rep is likely to cause very significant problems, something as high as 500 would almost certainly literally kill the whole site. If it's low enough to not be a big problem, say 5, then it doesn't really help, either. –  Servy Oct 21 '13 at 17:49
1  
@Servy And I didn't say that you're obligated to have a better proposal. I agree that 100 or 500 is too high, but what about 5 or 10? Wouldn't that successfully filter out the horde of "mindless zombies" that just dump their code into the text area and don't even bother to clarify the issue when asked? –  BartoszKP Oct 21 '13 at 17:56
    
Actually, you did: "but is there anything better you have to offer to actually solve the problem?" As I said, if you set the bar low enough then you just won't deter very many people at all, not even those who will ask poor questions. Though even with a limit of 5, you'll still deter quite a lot, and many of them will be quality questions as well as low quality questions. All of my previous points still apply, they just won't be as pronounced of an effect. –  Servy Oct 21 '13 at 17:59
    
@Servy Actually, asking if you have a better idea is not claiming that you're obligated to have one :) Ad rem: this is just guessing the future :) Perhaps some data explorer query would give our argument a more concrete basis. I'll try to do that. In general: seems that we at least agree that the issue needs to be tackled. –  BartoszKP Oct 21 '13 at 18:04
    
@Servy Still downloading the data through SE API (data explorer isn't enough unfortunately). It will take some time because of the throttling. Just letting you know that I'm still on it :) –  BartoszKP Oct 27 '13 at 17:58
    
@Servy I've added some statistics. –  BartoszKP Dec 9 '13 at 21:15
    
Where does the number 600270 come from? Is that the total number of questions asked in that time period, or the total number of questions that are from new users? You should be using the latter, but the number looks more like it's the former, in which case, it causes a misleading percentage. –  Servy Dec 9 '13 at 21:27
    
@Servy I've updated the description. I get your point, I'll calculate also the number of good&fresh questions. However this was to answer "how many good questions would be lost" regardless of their source. –  BartoszKP Dec 9 '13 at 21:31
    
Currently your "number of good questions lost" number sounds plausible (although if you included the queries you used, we could verify that). But the percentage is very misleading. I could just as easily say that the good questions asked by users with more than 100k rep only makes up [some very small percentage] of the questions asked, so they shouldn't be allowed to ask them. That simply doesn't follow. If 90% of the questions asked by those users are bad, that means something, but the stat you're currently showing can just indicate you're looking at a small demographic. –  Servy Dec 9 '13 at 21:38
    
@Servy Good call, the percentage was not the one I intended for. Now it's fixed. I've included the queries - sorry for the formatting, code blocks don't seem to work. Hopefully the SQL is simple enough though. The reasoning isn't that [if there is small percentage of something] -> [we should get rid of it]. It's rather [if the close vote queue was reduced by 23%] and [the cost of doing that is loosing small percentage of something] then maybe we should consider it. I've added also another idea - perhaps it should be posted as another post, as here everything becomes a bit cluttered. –  BartoszKP Dec 9 '13 at 21:55

At time of writing, there are 120K+ close vote questions.

Seeing as something drastic needs to be done, there need to be more harsh actions taken, which could looks as follows:

  1. Automatically close/delete questions that receive a certain negative score (say -3 or -5). This reduces the time/workflow required to close vote poor questions.

  2. Give some kind of reward when downvoted questions are eventually closed. It should be in the form of reputation, because if a number of people agree together that a question is bad, they have helped clean up the site by removing it. Maybe give each user that helped close the question by downvoting it +1 or something.

By way of example, for point 2, if you helped close 20 questions by downvoting it, you get 20 extra reputation, which is not exhorbitant. It's just enough to say thanks, but not so much that people will start closing questions that are actually valuable.

You should perhaps then increase the reputation required to get the downvoting privilege, because to someone with 125, downvoting will look like an easy way to gain reputation.

Sure, you don't get all your stats about why stuff was closed, but things are too desperate now to worry about that.

share|improve this answer
    
Downvoting questions does not cost anything. –  hims056 Feb 21 at 7:43
    
Not anymore? It used to. I haven't downvoted anything in a long time. –  mydoghasworms Feb 21 at 7:43
    
Mmm, I guess you're right. I will change my answer. –  mydoghasworms Feb 21 at 7:47

To reduce the backlog:

Gamify closing more.

Right now people review the close votes queue to get badges and for the love of the site. If that's not sufficient to clear the queue, add more incentives. SE is an excellent example of how well gamification can drive participation in an otherwise not very exciting activity. Why not apply the same principle here?

Obviously this will require some careful thinking to avoid incentivizing undesirable behavior, but that's the sort of tinkering that SE requires generally so it's nothing new.


One way to gamify the close queue without incentivizing bad behavior. This is meant more as a proof of concept than a completely foolproof system, something to distinguish between incentivizing well and incentivizing poorly.

Each close vote cast earns 1 rep for the caster if the question is subsequently closed. Each Do Not Close vote earns 1 rep for the caster if the question reaches five DNC votes before reaching five close votes (i.e., being closed). If a caster is on the wrong side of the majority, voting DNC on a question that everyone else votes to close, the caster loses a point. Reviewers aren't shown others' votes to avoid blindly voting with the majority.

In this system, there's a reward for reviewing questions that should potentially be closed, but only if other users agree with you. This is meant to be exactly analogous to the general SE model in which questions and answers are rewarded if and only if the community judges them useful.

share|improve this answer
25  
You do realize that they did this for the other queues, and it's ruined them. I literally used to spend at least an hour a day reviewing content, now I won't even go into them because too much of what I do is overridden by people not actually reviewing content. Close vote reviewing is the only queue for which this currently isn't a problem. –  Servy Oct 18 '12 at 18:58
5  
@Servy That's not a problem with gamifying the queues, it's a problem with doing it poorly. Maybe it's impossible to incentivize closing correctly, but I doubt it. –  blahdiblah Oct 18 '12 at 20:24
    
First off, I don't agree with the premise entirely, at least for certain queues. Even without any incentive there is clearly some people doing the work, and the quality will go down at least by some amount when you bring in people who don't want to do the work. At the very least, I'd wait until we have some more positive results with the incentives from other queues before rushing to implement more here. –  Servy Oct 18 '12 at 20:27
    
@Servy Couldn't you make that same argument for SO? –  blahdiblah Oct 18 '12 at 20:34
    
@Servy The problem with the other queues is that there's no review of the reviewers. Users can either actually review, which takes time, or just smack "looks good" which counts equally as a review and takes no time. –  blahdiblah Oct 18 '12 at 20:36
    
Yes, and how does that problem not apply to this queue? –  Servy Oct 18 '12 at 20:37
    
@Servy See update. –  blahdiblah Oct 18 '12 at 20:48
3  
@Servy: Solution: make a review review queue! –  Wug Oct 18 '12 at 20:49
7  
@blahdiblah Your solution is substantially worse than the problems created by the incentives in other queues. The problem is that the bad reviewers are all consistent; they do the same thing regardless of the content of the post, this means that they always agree with each other. If there are a lot of poor reviewers (in many of the other queues they will outnumber the quality reviewers, at least at certain times of the day) it is the good reviewers who will be punished by the system, not the bad ones. This will only encourage good reviewers to agree with the bad ones, feeding the problem. –  Servy Oct 18 '12 at 20:52
    
@Servy I added a potential fix for this problem, but that's not really the point I'm trying to make. Gamification works. That's what's driven SO's success. If we can agree on that, then I'm eager to try and find the way to do the gamification right, but I want to start from some common ground rather than having to produce the perfect system before the notion that gamification might work is even accepted. If you think gamification is doomed to fail for this queue, I'd be curious why you see the task as fundamentally different than SO generally. –  blahdiblah Oct 18 '12 at 21:14
    
See this question. In particular focus on the most highly upvoted answer and my answer. –  Servy Oct 18 '12 at 21:17
    
Oh, and not showing other's votes would make the system even worse. Seeing what other's have done is very helpful. Often I'll look at a post in the queue and not see a problem at first glance, but when I look at the reason another person voted to close, or reject the edit, I look for that problem and often find it. That is very useful for me as a reviewer. Again, as I said before, those not actually reviewing tend to just do the same thing for every item without reading it (i.e. approve all suggested edits, upvote all first/late posts, do not re-open) so they all agree (inadvertently). –  Servy Oct 18 '12 at 21:21
    
@Servy I think that Nicol Bolas' answer on that question is essentially correct, we just draw different conclusions from it. We all agree that incentive without review is a problem. He, and you it seems, think the solution is no incentives. I think the solution is to add some sort of review, ideally implicit in the form of voting instead of explicit like a review review queue. –  blahdiblah Oct 18 '12 at 21:45
    
@blahdiblah And I think that such voting wouldn't work because the number of poor reviewers outnumbers the number of quality reviewers, so the bad reviews would end up punishing those doing it correctly. If the number of bad reviewers were much smaller then a consensus-based solutions such as yours could actually work. –  Servy Oct 19 '12 at 19:35

Not the answer you're looking for? Browse other questions tagged .