186

I know this question is going to make me unpopular here – but I feel I have no other options.

I have been concerned by the size of the close votes review queue for quite a while.

When I started reviewing (less than a year ago) this queue already had ~30K questions pending review. Time passes and the queue only gets bigger. It is close to 95K now and increasing all the time.

This queue size issue has troubled many users, that raised flags, proposed changes and asked questions. Yet it seems like SO dev team is doing close to nothing to handle this pressing matter.

Therefore, I see no other option but to go on a strike: I will not review any close vote question until a serious effort will be made by SO dev team to address the close vote queue size issue.

Where do you stand on this matter?


Making this question more concrete:
The vote-to-close queue size has actually two issues:

  1. A one-time effort to bring its size to (or very close to) zero.
  2. A long-lasting solution that will reverse the trend of more questions entering the queue than leaving it, thus keeping this queue at manageable size.

Without addressing the second issue, any solution that covers only the first one will be only a temporary patch and will not last long.


Edit/Update: I do not ask SO dev team to drop everything they are doing and work 24/7 on this issue. However, I do expect to get some long-term attention of the team: to take the time to implement one (or even several) solution(s) and follow up on them to see if they have the right impact.


What's up? (Dec 3rd,2013)

I started my strike two weeks ago with queue size ~95K. Today it hit 100K!

enter image description here

A lot of words were written (and put on hold/marked as duplicate) questions were asked, answers were given - but we see little action actually taking place to reverse this worrying trend, the steady growth of the close-vote review queue.
So, I see no reason to quit my strike just yet...


A note regarding marking this post as duplicate:
Well, I looked into this question, apparently it is also a duplicate of this question. All these questions were asked to raise the flag "Hey, we SO users are CONCERNED by the HUGE queue size". Since no significant change has been made, and the queue is just getting longer, this red flag needs to be raised OVER and OVER again. Closing these questions only sends the message "we don't care about this issue" – which judging by the lively discussion here is not the message you want to send.
So please, consider not closing this question as a duplicate of those questions.

61
  • 15
    -1 because the whole review system was created to help us review faster and easier. Changing it isn't a trivial thing and the team got other things in their hands. And they do more than nothing: filter option was added, so user can see only posts in certain tags or close reasons; autits were added to filter out robo reviewers; and probably more I'm not aware of. Nov 20, 2013 at 15:56
  • 23
    I'm in - since May 29 :)
    – gnat
    Nov 20, 2013 at 15:57
  • 70
    @ShaWizDowArd - the whole point of reviewing is to make SO content better. And this queue size issue clearly shouts "something is NOT working here"
    – Shai
    Nov 20, 2013 at 15:57
  • 13
    This strike seems counterproductive to me - it means one reviewer less. I'd rather look for a proposal that I believed could solve the issue, and put a bounty on it. Nov 20, 2013 at 15:59
  • 27
    @S.L.Barth I believe there have been quite a few, unresponded to. Nov 20, 2013 at 16:00
  • 46
    @ShaWizDowArd but of course! tiny black ribbon is soooo much more important than 90+K questions that are worthy of either closing or keeping open
    – gnat
    Nov 20, 2013 at 16:00
  • 10
    Or it could be a symptom of people not reviewing close votes because the queue is so large. The nice cycle of it being to large to review and growing larger because its not being reviews as much.
    – Joe W
    Nov 20, 2013 at 16:03
  • 9
    @ShaWizDowArd you are assuming that most questions in the close queue are old questions that can wait a few days more. This isn't the case. Nov 20, 2013 at 16:03
  • 39
    My guess is a LOT of people have stopped reviewing - at least informally.
    – user206222
    Nov 20, 2013 at 16:09
  • 23
    @JoeW don't know if you aware or not but attitude set at SO (the largest site) appears to spread to smaller sites and damage these. Askers used to having their low-quality questions fly without problems at SO, go to other sites with same kind stuff and complain "oh why do you close stuff that goes so well at SO". That's a network wide damage, don't hope it's contained within SO
    – gnat
    Nov 20, 2013 at 16:09
  • 20
    @S.L.Barth proposal to fix hotness formula got 9 bounties from 9 users, how much bounties would you recommend to get attention to CV queue size?
    – gnat
    Nov 20, 2013 at 16:13
  • 9
    If everyone goes on strike how long till the queue breaks 1 million flags?
    – Joe W
    Nov 20, 2013 at 16:17
  • 15
    @JoeW exactly the same time as if noone goes :-) Nov 20, 2013 at 16:17
  • 8
    @ShaWizDowArd that is not the suggestion, however. I disagree with the strike, but I do want more dev team attention to the queue. Nov 20, 2013 at 16:23
  • 8
    I also stopped after reaching around 3.5k reviews in the close votes, raised the issue here on meta a few times, also made some proposals, but no response from SE. The most disturbing part is the no response from SE part...
    – SztupY
    Nov 22, 2013 at 13:12

24 Answers 24

207
+650

There is a fundamental fallacy the employees at Stack Exchange are making.

Find ten people for a beach cleanup. Tell them they're going to clean eight miles of beach. Chances are, they'll say "Okay, that sounds reasonable. Might take a couple days."

Find one hundred people. Tell them they'll clean eighty miles of beach. Chances are, they'll say "That sounds difficult." A handful of people might leave. Has the work per person changed? No.

Find two thousand people. Tell them they'll clean the entire West Coast of America (~1600 miles). Chances are, most of them are going to say "No way in hell we'll be able to do that." Has the level of work per person changed? No. (also worth noting: does it help when people leave? Definitely not. But they will anyway.)

So, what's the point here? People look at one hundred thousand questions, and people begin to go "there's no way in hell this will happen." So they leave. So it takes longer, and the deficit we grow becomes worse until nobody is left reviewing. Yeah, you can say that "it's for you, not us," but so is a beach cleanup.

If you want people to tackle a task, give them the tools they need to do so. Make the task seem doable, and people will come back. This is the problem with Stack Exchange's approach. Even if people are doing it for themselves, they'll leave if the task appears insurmountable.

My suggestion to Stack Exchange is: address this. I'm not going to create a new idea because there are at least four dozen out there. But the issue needs to be addressed.

37
  • 10
    You say that, and yet a year ago the very low quality question queue had 50 thousand items in it. It was cleaned out in just a few months. Nobody, at all, complained about the size of the queue; people just did it.
    – Servy
    Nov 20, 2013 at 17:21
  • 36
    @Servy The size of that queue wasn't growing. It didn't look insurmountable. Case in point: progress was visible. Contributions are apparent. The CV queue doesn't have this: it continues to grow, and people continue to leave.
    – user206222
    Nov 20, 2013 at 17:22
  • 12
    What is your suggestion, then? Nov 20, 2013 at 17:23
  • 12
    @Jan My suggestion is that SE actually address the problem, instead of passing it off as a problem with the reviewers, or something that doesn't need addressing. My point is that StackExchange isn't addressing this, and they need to.
    – user206222
    Nov 20, 2013 at 17:25
  • 1
    @Servy But then the size of the queue began to go down. The one number indicative of progress indicated progress. Also not happening with CV.
    – user206222
    Nov 20, 2013 at 17:25
  • 4
    @Emrakul What problem? The queue being large isn't a problem, in and of itself. People thinking that it's a problem despite it not being a problem, is the only problem that I see.
    – Servy
    Nov 20, 2013 at 17:26
  • 8
    @Servy It's not a problem, in and of itself. It is a fallacy to look at that and say "nothing needs to be done." The psychology of the reviewers driving the queue is just as important as the reviews themselves. It's the people that do the reviewing. And it's the people who think they are making no progress.
    – user206222
    Nov 20, 2013 at 17:28
  • 4
    @JanDvorak - How about two queues? One for recent close votes and another for more ancient ones. If the "recent" queue is kept cleared then the definition of recent could slowly get extended to make inroads into the older flags. Nov 20, 2013 at 17:28
  • 77
    +1 - I don't know about others, but I've experienced this. I jump in, review a dozen or so questions. The next day, the queue is bigger, so I jump in, do what I feel is my part. Next day, it's bigger and then I think to myself "You know, my effort here, which really earns me nothing on the site, isn't making a lick of difference. I'm going to go do something more rewarding."
    – JDB
    Nov 20, 2013 at 17:31
  • 8
    @Servy, "Nobody apparently has an actual problem with the review queue itself though. The only actual solutions to the problem are things that people don't want to actually have happen." Well, I'm one data point that proves your absolute assertion incorrect. I have a huge problem with the close vote queue, and personally would love to see nearly any of the proposed solutions implemented. You don't speak for everyone when you make statements like that. In fact, there is a large portion of people that clearly disagree with you.
    – Ben Lee
    Nov 21, 2013 at 20:07
  • 2
    @Servy, and regarding your point that cv review rates have not gone down, that's not particularly important. When the cv queue start growing faster, the idea is to also get cv review rate to increase. And it's not increasing because this huge wall is psychologically holding people back. Holding back people like myself -- I don't ever do cv reviews, but I certainly would if any number of suggestions were implemented.
    – Ben Lee
    Nov 21, 2013 at 20:10
  • 1
    @Servy, I'm not saying that the actual numbers are decreasing. I'm saying that the numbers are not increasing to keep up with the increased rate of queue growth, and my assertion is that they would be increasing if the tooling were better. It's not an actual loss in reviewers, it's an unmet potential -- people that are not currently reviewing that would be, or people that are currently reviewing that would be doing more.
    – Ben Lee
    Nov 21, 2013 at 20:24
  • 1
    @Servy It implies that there are more people able to review, but they aren't.
    – user206222
    Nov 21, 2013 at 20:27
  • 5
    @Servy, all the evidence I have is the dozens of people who have posted questions/answers asserting things along this line (I am myself an example of an unmet potential psychologically blocked by the queue qsize), and the hundreds of upvotes they have collectively received. If you want to call that "a handful of anecdotal claims" feel free. I don't think this can be as neatly scientific as you're trying to make it out to be.
    – Ben Lee
    Nov 21, 2013 at 20:28
  • 8
    @BenLee I think it is safe to say that 90 upvotes collected by now by this very strike announcement build a pretty solid evidence to support your point. I have only seen comparable level of disappointment in a famous complaint about review abuse
    – gnat
    Nov 27, 2013 at 12:33
52

Therefore, I see no other option but to go on a strike: I will not review any close vote question until a serious effort will be made by SO dev team to address the close vote queue size issue.

If, up until now, you were reviewing close votes to make us happy... You were doing it for the wrong reason. The close queue isn't for us - it's for you. If you don't see the point in reviewing questions nominated for closure, then nothing I say or do is going to change that.

That said, I understand the frustration involved in trying to use tools that don't seem adequate for the task. And I take full responsibility for not communicating what we're hoping to do to improve this tooling; it's been on my list for a while, but other matters keep intruding. I should note that we have posted answers on several occasions outlining areas where we're investigating changes; the problem is that we've nothing positive to report as a result of these investigations yet... At some point, just saying "we're working on it" starts to sound hollow, particularly when visible evidence seems to contradict it.

Never the less, here's another post on the matter; let me know if it helps you feel any better.

68
  • 44
    First of all: I am not reviewing to please moderators/SO dev team or any other supreme being. I do it because I want to contribute to the quality of SO community. However, when I feel that my good will and efforts are not appreciated - this is the moment I quit.
    – Shai
    Nov 20, 2013 at 16:47
  • 17
    @Shai appreciated by whom? You should be satisfied by crap being closed. Nov 20, 2013 at 16:48
  • 7
    @Shai How does the size of the queue affect how you're appreciated? That a queue says there are lots more items to review doesn't mean your help isn't appreciated. That a queue is empty doesn't mean your contributions are appreciated. Appreciation has no correlation to queue size.
    – Servy
    Nov 20, 2013 at 16:48
  • 7
    @Servy it is a good feeling to know you can keep up with the crap. If the queue is growing, reviewers can get the feeling they don't keep up with expectations. Nov 20, 2013 at 16:50
  • 26
    @Shai, as I said, I completely understand the frustration that shoddy tools can cause. That said, please be careful when you seek appreciation: I've closed far, far more questions on Stack Overflow than you, and not once have I ever been thanked for it. Called a Nazi? A pedantic jackass? A blockwart? Yes, in spades. But never thanked. If you're not doing it because you believe strongly that it is a good and right thing to be doing, you're going to find yourself unhappy regardless.
    – Shog9
    Nov 20, 2013 at 16:51
  • 3
    @Servy: The SE team and ourselves should be working together to clear the close-vote queue... the SE team by giving us more effective tools, and us by attempting to clear the queue. We're doing our side of the bargain, and when SE aren't doing theres, it's pretty frustrating.
    – Matt
    Nov 20, 2013 at 16:52
  • 8
    @Servy: An empty queue is more effective than a full one, because the queue has dealt with all the problems. Whatever they are, and however many they are. The close-vote queue is not dealing with all the problems, unlike all the other queues on the site, making it ineffective at what it was designed for.
    – Matt
    Nov 20, 2013 at 16:58
  • 3
    @Servy A large/growing queue is a sign that the queue is insufficient. That it is not doing it's job sufficiently well. There are lots of ways you could make it worse, such as by not allowing new items into it that would mask this symptom but it is a symptom none the less. Similarly you can mask a person running a fever by dumping them in ice water; it does not mean you ignore the symptom or say they'd be much worse off if their temperature was 0C Nov 20, 2013 at 16:58
  • 12
    Btw @Shog9 thank you for everything and I'm certain (within experimental margins of error) that you are not a Nazi Nov 20, 2013 at 17:04
  • 2
    This approach won't get us anywhere; I've posted why I think so in an answer below.
    – user206222
    Nov 20, 2013 at 17:19
  • 7
    @Servy certainly not. The key is to get the reviewers to scale to the magnitude of the queue. A successful shopping line isn't one that never has customers - its one that is able to handle all the customers and has slack in it to pick up spikes if needed. If the rate of things in the line is going up faster than more cashiers are being put on duty, thats a failure of the system, not a success. Likewise, if something that should be migrated sits for 60 days before it gets 5 votes, thats a failure. If a bad question gets a bad answer a week after asked - before its closed, thats a failure.
    – user213963
    Nov 20, 2013 at 20:56
  • 7
    @Servy ... those are things that diminish the quality of the site. Migrating questions in a timely manner so that they can be addressed on the proper site - thats good. Having a poorly focused question get closed, fixed, and reopened in short order - thats good. Getting poor questions closed quickly before they are answered so that the poster learns what is and is not acceptable is good (before they ask 6 questions, get 6 answers, get 6 closed questions, and get a question ban). Closing questions is a key feedback mechanism from the community to the poster - delaying it hurts that cycle.
    – user213963
    Nov 20, 2013 at 21:02
  • 3
    @Servy I am saying that a queue that is growing in length for things to be handled is detrimental to the quality of the site and impacts the quality of other sites (people reposting questions that don't get migrated timely). That is the key point I am trying to communicate. The front end manager should call up more cashiers to work - when we are volunteers it means adding incentives or finding ways to help people do the reviews in a less painful way (notifications akin to the suggested edit review, easier filtering, etc...).
    – user213963
    Nov 20, 2013 at 21:14
  • 4
    @Servy This question isn't about proposing ideas, it's about getting a response to some of the existing proposals Nov 20, 2013 at 21:50
  • 3
    @Servy while I could be drastically wrong I am of the understanding that the two major reasons that people don't partake in the close vote review queue for people who are otherwise active in the community for the site are: (1) the painfulness/not fun nature of doing reviews and (2) a philosophical disposition that things need to remain open. Failing that, I have difficulty understanding why active community members don't do reviews. If you have any insight that would helping me gain understanding into the minds of those people, I really would like to hear it.
    – user213963
    Nov 20, 2013 at 22:55
39

The problem is that the team might already work on a solution but does not communicate that.

We had a ton full of proposals how to reduce the close vote queue. But we do not seem to get responses to them. Not even in the form of a [status-*] tag.

So please, let us know what is planned - even if nothing is planned right now. A status update would be really great.

Update

There is the feedback of our community manager. Thank you.

8
19

This is largely a copy of a previous answer of mine, with updating it for the current situation, and with recent experience.

Let's take a look at the problem. As of today, this queue is up to 95.8k questions in the queue. Assuming each has 1 closed vote, that means that a total of 380K actions are required to close all of the questions. Given 40 per day, that means that almost 10K new users with at least 3K reputation would have to clear out 40 in order to close the queue. Then after it is closed, then more people need to review them than have been doing so. So, how can these get reviewed? There are a few ways:

  1. Increase the number of people reviewing.
  2. Decrease the amount of things that need to be reviewed.
  3. Increase the number of things that can be reviewed per day per person.
  4. Decrease the time per review required

Okay, given these options, how could any of them be done?

Increase number of reviewers?

Close vote reviewing is a painful, thankless task. It largely goes almost unnoticed, except for the occasional badge. Sure, it's the easiest badge of the various custodial badges to earn, but a closed vote decision can be difficult sometimes, depending on the nature of the question. Perhaps there should be some incentive to encourage more reviewers, either in the form of reputation, or increased privileges.

Also, make the task seem more manageable will improve the likelihood that it can happen. Show the flags with a given tag. Let people see that for their tag of choice, the number of flags is going down! That will encourage them to try harder, and make a difference!

Decrease the number of things to review

There are a couple of ideas that I have that fit into this category. Either somehow things need to be automated better, or the number of close votes could be reduced. As the latter could be dangerous, perhaps we should just allow certain users to have more closed votes, perhaps those with a very high reputation, or maybe those who have reviewed more closed votes overall. Both help to filter out the relatively new closed voters, and leave those with more knowledge of how the site goes more power to help it out.

Increase the number of things that can be reviewed

This improves the situation so long as you don't burn people out. As the pile is getting deeper, it could be a quick short term solution.

Decrease time per review

Filtering by tag or closed reasons, better displays to help users figure out what's going on, anything to make the task of reviewing easier increases the likelihood that a user will use all of their closed votes per day, and come back tomorrow and use them too.

Proposed Solution

I'm sure there are other ideas out there, but here's my preferred method:

  1. Allow users with a Reviewer Badge to get 2 close votes, and with a Steward to get 3 closed votes (Might need to be tweaked, but I'm putting out something here)
  2. Increase the number of reviews per day for the closed queue, at least for a while.
  3. Do some filtering automatically. The top questions I look at should be for tags I am familiar with. This will make the review queue simpler to manage, as I won't have to learn what the question is about before I look into it. I know that I can filter them manually if I want to, but it's not that obvious to a new reviewer, and thus I think it should be done automatically. This will help people to feel more comfortable reviewing the questions.
  4. Some tweaks could be made to make reviewing closed questions easier.
  5. Show the number of closed questions with a given tag. That will help people to think they are making a difference, encouraging them to come back more frequently.
  6. Give physical rewards to people for taking such a massive undertaking. After all, closing questions is a very important part of SE, as moderators are, why not reward users for taking on such a task? Perhaps a gold badge user will be given something?
3
  • 6
    #3 especially. Non-robo reviewing is hard work, and much harder for topics with which I am not familiar. Once I stumbled on the filter, so that I can review topics I know well, reviewing became, if not easy, at least much easier, and at times even rewarding. Feb 11, 2014 at 15:09
  • 5
    Also agree with #3. There are questions that anyone can tell are bad, but then there are questions that might look good at a glance, but an expert in that field (tag) would be able to look at them and immediately say "nope". Perhaps anyone who casts a close vote for the first time should get an info popup saying "Use the filters Luke"?
    – Ian Kemp
    Feb 11, 2014 at 15:42
  • 1
    Tag filtering is a nice example how to make this all more bearable. For example, it certainly doesn't make any sense to show ignored-tagged questions unless the queue is almost empty. I suppose that heurestical analysis of what is easy for me to close based on my tag scores would be complicated, but first steps could surely be done. However, this all means that the queue is basically different for each user -- still, storing like 50 first items for each reviewer shouldn't be so difficult, what's difficult is probably to make the list. +1 for #5+6 too. However, #1 seems just too dangerous.
    – yo'
    Feb 14, 2014 at 18:07
19

[Note From The Future]

This discussion moved forward since I posted this and my views have evolved. Originally, there was some conflation of the review queue size and the number of questions that needed to be closed. In reality, many of those questions don't need to be closed - one or two users out there didn't like something about the post and thus flagged or voted to close. This was addressed very well by Shog:

Regarding the Stack Overflow close review queue

See, the problem with a lot of the suggestions floating around here right now is that they make a couple of shaky assumptions:

  1. Most of the questions in the queue actually need to be closed.
  1. A handful of people working REALLY HARD could close them all in no time, if we just gave them more privileges / required fewer close votes / skipped the whole "review" thing and just closed them all automatically / etc.

#1 I'm just not seeing. Oh, for sure there's a lot of crap in there... But there's also a lot of stuff that's just in the queue because someone didn't know what the hell they were looking at and decided to flag it, or thought "minimal understanding" meant "already solved the problem and is just posting here for typing practice". Especially once you get outside the PHP tag.

#2 is true in theory, but... We've kinda been trying to move away from that - the big hope for review was that it would empower folks enough that we wouldn't need 15 moderators closing stuff all day long to keep up. And the truth is, it's a lot harder to review stuff when you don't know jack about the topic.

My current opinion is that the size of the review queue is not really that big of a problem. Most questions that are going to be closed are already closed in relatively short order. For example, 80% of all questions that were closed in 2013 were closed within 24 hours of being posted. The remaining questions that ended up being closed were closed not long after that - nearly all of them within a few days.

To me, that's a strong signal that the questions which need to be closed are being closed. The rest - well, the community clearly doesn't see a need to take action on them, so they hang around in the review queue until the votes expire (or, if the question is actually good, achieves enough "do not close" votes). While that may produce an ugly looking number, the reality is that there doesn't appear to be much of an underlying problem with the queue.

[/Note From The Future]


I've suggested something similar before, but, personally, I'd be much more motivated to tackle the close-vote queue if I could get a little bling. Maybe a fancy hat or pin on my profile picture saying something like "I voted".

Others, seeing my awesome bling, would ask how they could get some. I'd say, "just review some close votes" - and off they'd run to get their own bling.

It'd be relatively easy to implement (it was already done for Christmas last year - code must still be lying around somewhere) and would act as both an incentive and as an advertisement.

Bling could have an expiration date. Review 10 posts in the close vote queue and you can pick a hat. You keep the hat for 24 hours, then it expires. Review 10 more posts and you can pick another hat. Review 20 posts, and you get a bigger selection of hats, etc. Review 30 posts, and you can have a magic unicorn horn! (Oh the possibilities!)

The bling need only be part of an "initiative". Meaning, once the queue falls below 10k or 5k, the hat reward is retired. Any time the queue starts to grow again, hats could be turned back on.

Looks like I'm not the only person who thinks this is a good idea: Provide incentives for reviewing close votes in upcoming Winterbash 2013

In seeking to educate at the same time that we reduce the queue, what if we also had some special hats for using close queue features, such as a hat for applying a filter, etc. (I love "etc."! Frees me up from having to work out the details myself.)

2
  • 3
    but, wouldn't more bling cause more roboreviewing and lower quality reviews? Nov 20, 2013 at 17:45
  • 7
    @JanDvorak - Moreso than already exists in the pursuit of badges? Possibly - but then again, we have mechanisms for dealing with robo-reviews already. Fail a couple audits - lose your hat priveleges for the day. I think I'd rather try it first and see if it's failing then not try anything.
    – JDB
    Nov 20, 2013 at 17:49
12

Why not to adapt the limit of review per day according to the queue size like this (it is an example):

when the "close vote queue" reaches {20k}

  • people with rep between 3k to {9999} = {60} reviews / day
  • people with rep between {10k} to {49'999} = {100} reviews per day
  • people with >{50k} = {150} reviews per day

When the queue is lower, just come back to the actual rules.

Numbers between {} can be adapted by the dev team according statistics or just rule of the thumb.

2
  • 17
    40+ close reviews per day is enough of a slog as it is - that would only lead to less care and attention being paid, and/or more burnout. We don't need more effort from existing 40-per-day reviewers, we need more reviewers. Nov 23, 2013 at 14:25
  • In theory, yes because you assume the 16'000 people having more than 3k rep are involved in reviewing: it means more than 600k review tasks can be done every day. But the reality shows it is not the case. Otherwise, something else have to be made to motivate the other 3k+ users.
    – рüффп
    Dec 8, 2013 at 11:15
7

You said:

I will not review any close vote question until a serious effort will be made by SO dev team to address the close vote queue size issue.

Well, now a serious effort has been made. I hope you'll end your strike and join these guys (and gals) in their flamethrowing shenanigans!


Update five days later:

Hooray, you've ended your strike! With, lets see...uh, well...it looks like...ah...one review.

???

1
  • 1
    I never was a "mass reviewer", but now I got back to occasionally review close votes.
    – Shai
    Jan 28, 2016 at 14:49
6

I review on occasion. Enough to watch the queue constantly grow at least.

However, there is a massive undertaking going on at a constant rate for closing questions so I think undermining it is a terrible idea. A strike? No thank you.

Does something need to be done? Perhaps it could be made more efficient.

How? Lets break down the way the feature works.

Goal: Close all the bad things.

Requirements: Bad thing + 5 consenting votes of badness.

Priority: Address the newest thing that may be bad.

To me, this isn't the best way to organize the bad things. Some of the newest things may not be bad. However, some of the things with 4 consenting votes are more than likely bad.

Suggestion: Address the things which have the highest chance of being bad.

1/5 consenting votes is a low percent of being bad. 4/5 consenting votes is a high percent of being bad.

The priority of the review queue should be aimed at reviewing those questions with the highest amount of close votes first.

It could be built in, or perhaps just part of the filter feature. Sort by: "Newest","Oldest","Most Close Votes","Least Close Votes".

16
  • 2
    Questions with multiple previous reviews are already prioritized in the queue. This doesn't necessarily equate to things with a lot of votes - but it's usually pretty close.
    – Shog9
    Nov 20, 2013 at 16:53
  • @Shog9 - I guess I just don't see much evidence of that. If you look at the review queue history, a majority of the recent actions are on questions which were posed within the past hour or two.
    – Travis J
    Nov 20, 2013 at 16:54
  • @Shog9 I thought they were sorted by the last vote? If are sorted by the number of past reviews, then by age, then great! Nov 20, 2013 at 16:55
  • @Shog9 - For example (from the past 10 minutes): stackoverflow.com/questions/20098128/…, stackoverflow.com/questions/19887952/…, stackoverflow.com/questions/20090552/…, ... well this is getting tedious already... etc.
    – Travis J
    Nov 20, 2013 at 16:57
  • @Travis: there are a few things that make this more complicated. 1) there's a huge backlog with no reviews at all. 2) the question you're given for review is chosen semi-randomly from among (a sizable chunk of) the most recently-enqueued tasks, 3) if you're doing a lot of reviews, you'll blow past already-reviewed questions pretty quickly. However, most of this doesn't matter if you filter by a reasonably specific tag...
    – Shog9
    Nov 20, 2013 at 17:04
  • @Shog9 what if the queue size backlog was shown broken down into how many questions are how close to being completed? Nov 20, 2013 at 17:07
  • @Shog9 - Filtering by the c# tag, it took 20 skips to get to a question for review which was not asked today.
    – Travis J
    Nov 20, 2013 at 17:19
  • 1
    @Jan - To review a majority of questions asked today? New questions get a ton of attention from the active and newest list and they get reviewed with that attention as well. However, questions which have had 3 close votes sitting for the past month get no attention. How is that "sorted pretty well"? I do not agree. Reviewing something from 45 minutes ago with 1 close vote as a possible duplicate instead of reviewing a question with 4 close votes which was actioned a month ago (and if voted will be removed from the queue) would be sorted much more efficiently.
    – Travis J
    Nov 20, 2013 at 17:28
  • @TravisJ you are right. I retract and delete my comment. Nov 20, 2013 at 17:30
  • Try a smaller tag, @Travis - entity-framework perhaps, or even asp.net-mvc
    – Shog9
    Nov 20, 2013 at 17:31
  • Moreover, it wastes users time to constantly review questions asked same day because those questions are subject to change and improvement. Why should any high rep user who could be answering questions spend an hour or more reviewing until the daily limit when a majority of the content they review will either be improved or would have been closed by users coming across that question in the active/newest list.
    – Travis J
    Nov 20, 2013 at 17:31
  • @Shog9 - Same pattern with EF, but since the frequency of those questions is lower, the queue starts with a question from yesterday. In fact, of the 20 I skipped there, almost all of them were asked this week. I still see a pattern of recently asked being given priority.
    – Travis J
    Nov 20, 2013 at 17:36
  • @Shog9 if they are sorted semi-randomly, is any priority given to migration questions where there is a time limit on when an action for it can take place? Arguably, some of the "belongs on" comments come from an inability to get things migrated in a timely fashion after it drops out of visibility (which leads to other problems - yea, everyone has their pet peeve).
    – user213963
    Nov 20, 2013 at 17:40
  • Something may be broken there then. Thanks @Travis.
    – Shog9
    Nov 20, 2013 at 17:40
  • No special priority is given to migration votes, @MichaelT. Keep in mind though, I like "belongs on" comments as long as they're reasonably accurate - gives folks a chance to look around a site before they end up with a question there.
    – Shog9
    Nov 20, 2013 at 17:42
6
  • Force new users to take and pass a quiz on the FAQ before they can post any questions;
  • or require a certain number of accepted answers before being able to post a question;
  • or require a decent amount of rep before being able to post a question (and I've suggested this before).

Then we'll have fewer crap questions, and less to review.

Doesn't help with the backlog, but I think it'll attack the more fundamental issue here which is that SO has scaled out of hand. There's "being nice to new users", and there's letting yourself be walked all over.

14
  • 1
    Why not give a smack on the head for any user who posts crap questions? You are right it would be a better world if new users (and some of the old ones) will actually take the time to read the FAQ BEFORE posting... I just can't see this happening. This quiz will more likely repel many users who otherwise may contribute a lot to SO.
    – Shai
    Nov 27, 2013 at 12:19
  • 6
    @Shai: If they're not willing to prove that they know how to contribute constructively, I doubt that they'd end up doing so to any degree that means we'd miss them. We need to stop being so nice. An alternative is to require a certain number of accepted answers before being able to post a question -- this is a community, not a free solution-solving service where everybody must be able to get their problems solved for them without needing to contribute anything back. So kind of a technical paywall. I'm sure this idea'll irk some people here but I really like it. Nov 27, 2013 at 12:45
  • I think the "neceness" of SO is a KEY feature of this website and community. I'd hate to miss this property.
    – Shai
    Nov 27, 2013 at 12:47
  • 2
    I prefer your second option. Will you consider "softening" it to require a certain number of upvoted answers before being able to ask a question? Or maybe just set a 25 rep privilige to ask a question?
    – Shai
    Nov 27, 2013 at 12:49
  • 1
    @Shai: There's being nice and there's allowing yourself to be walked all over by any of 7 billion people. Nov 27, 2013 at 12:49
  • 3
    @Shai: I would be satisfied with a higher rep barrier, yes. Nov 27, 2013 at 12:50
  • 1
    You are right, if we destroy the site entirely we will be able to reduce the close queue Nov 27, 2013 at 14:57
  • @RichardTingle: I enjoyed your strawman argument, thank you! Nov 27, 2013 at 20:41
  • @LightnessRacesinOrbit I feel calling it a straw man is to give it too much credit. It was more of a remark. Regardless, putting significant barriers in the way of new users joining the site is only going to lead to a much smaller user base Nov 27, 2013 at 20:49
  • @RichardTingle: I've seen no evidence that that would be a problem. Jan 26, 2014 at 15:05
  • 2
    My take on the whole "barrier to entry" thing: if a new user is really interested in becoming a part of SO, and not just a one-question-wonder, they most likely won't have an issue with jumping through a few hoops to be able to post a question. Rather set some standards than be too nice and get swamped by a tidal wave of crap.
    – Ian Kemp
    Feb 11, 2014 at 15:49
  • glug glug glug 🍌🍌🍌🍌🍌🍌 Feb 11, 2014 at 15:54
  • My suggestion: 1 accepted answer or 2 upvotes on your answers give you the right to ask a question. If you are good enough, should take less than 10 minutes to achieve. As an example, Jon Skeet earned 39 upvotes on his first answer on SO main. This will probably not filter all help vampires, but 99% of them I'd bet. Feb 13, 2014 at 19:42
  • @Neolisk: Sounds good to me Feb 14, 2014 at 1:53
4

The size of the review queue represents the number of users willing to start a closing process as they find bad things in their browsing, but unwilling to put some time into deliberately finishing a few.

I'm not sure what makes this a problem for the team or the moderators. It is a user thing.

Perhaps that queue needs more visibility, but that is the only change that leaps to mind.

13
  • 12
    Not really. The size is the number of questions with at least one flag. User can't finish such review item by himself, it takes 5 to close a question. Nov 20, 2013 at 16:24
  • 1
    @ShaWizDowArd Really. Each user that looks at a question makes progress. Maybe you can't finish any given one yourself, but someone has to cast the middle votes too. Nov 20, 2013 at 16:26
  • 3
    @dmckee He's referring to users without 3k reputation. They can add an item to the queue, but can't even cast a single vote. It was only when flags were directed to this queue that it began increasing in size.
    – Servy
    Nov 20, 2013 at 16:28
  • @Servy I think that also flags used to age once and now they don't. Right? Nov 20, 2013 at 16:29
  • More visibility is probably a good idea (get a balloon like that for the suggested edits, maybe?), but I don't think it's all that needs to be done. Nov 20, 2013 at 16:30
  • 1
    @ShaWizDowArd I believe I remember Shog mentioning something not too long ago about re-working how they are aged. Before they'd only go away when there were 100 views since it was cast, now it's a bit more intelligent about it, for example considering people viewing the post with 3k rep, rather than just those with any rep, but having a much lower limit. Not sure about the detail though, or if/when it was rolled out.
    – Servy
    Nov 20, 2013 at 16:33
  • @ShaWizDowArd I made a suggestion recently on how to treat (not handle) flags that lead to a question being added to the close votes queue: meta.stackexchange.com/a/202660/195761 (Basically by converting flags to votes once a valid vote has been cast). Not sure though whether to post is as a separate feature request to increase visibility or to add a bounty though... Nov 20, 2013 at 17:09
  • 4
    one of my own suggestions on how to encourage people to get into the review and increase its visibility - notify users of possible reviews on toolbar - this in part changes the "90k need review" to "40 need review" and that each individual can get that 40 to 0.
    – user213963
    Nov 20, 2013 at 17:28
  • @Servy considering views are stored as plain counter (no data who viewed what) that's unlikely, but do recall some other change indeed. Nov 20, 2013 at 19:58
  • @Spontifixus that's a dilemma indeed, good suggestion as many others! :) Nov 20, 2013 at 19:59
  • @ShaWizDowArd this post (thanks to shog's post above that linked to it) talks about it a bit (first bullet point). Not sure if anything has been implemented since that post, it seems that they had something in progress in September.
    – Servy
    Nov 20, 2013 at 20:00
  • @Servy thanks, didn't see Anna's answer before. But she did say "some ideas might be unworkable". :/ Nov 20, 2013 at 20:07
  • @ShaWizDowArd Yeah, my earlier comment was going off of vague memory. I honestly don't know for sure if there was any change to vote/flag aging, or, if anything, what was specifically done.
    – Servy
    Nov 20, 2013 at 20:12
4

Could a series of badges be introduced as an added incentive for continuous work on the close queue. This would be something similar to the "Fanatic" badge but with tiers similar to the "nice", "good", "great" question/answer badges.

  • Reviewed 40 close votes per day for 7 consecutive days
  • Reviewed 40 close votes per day for 30 consecutive days
  • Reviewed 40 close votes per day for 100 consecutive days

It could be that 40 votes per day is too much and if the purpose is to get people to get into the habit of working on the close queue every day, then maybe reviewing 20 (or maybe even just 10) close votes per day is enough.

1
  • I agree that 40 per day is too much and will result with too many "robo-reviewers" and this is something we do not like...
    – Shai
    Feb 13, 2014 at 18:48
3

I'm in agreement with many of the arguments here (notably that the enormous size of the review queue may be causing a vicious cycle of discouraging those who'd otherwise help clear said review queue), but to say that there's "no other option but to go on strike", seems a bit rash.

If there's individual apathy toward reviewing due to the queue seeming insurmountable by an unorganized mass of users (again, from the individual's perspective), then why don't we first see how we'd feel about collective, organized efforts? By the way, to those who've seen my proposal for the "non-competitive community event," this is not a plug for that idea; that idea requires an effort by the SE team, e.g. lifting the daily voting limits, the lack of which is OP's complaint to begin with. Instead, I'm suggesting we try something productive amongst ourselves, without needing to involve the SE team.

To those interested in forming some kind of a weekly reviewing collective, a "club" let's say, please join this Google Group (or the highest-voted alternative in the comments, in case Google Groups isn't what SO'ers prefer to use) please join this chat and leave a comment, and you'll be given explicit read access (as a way of getting a headcount).

Comment on this post with ideas about what this club should actually do. For example, "Let's all meet in a chat room at 10:00am every 6 or 8 days (that way it's not always the same day of the week, giving everyone a chance to participate), and review close votes together." If there's enough interest in such a club, then we'll go ahead and get started with the ideas expressed in the highest-voted comments.

* I'm adding the first 2 comments immediately so that they may be voted on and compared to alternatives.


tl;dr

Gather a roster of SO users who'd feel more enthusiastic about reviewing close votes in a group, and then TBD (via comments). Join this chat and leave a comment to get started.

9
  • 1
    To those in favor of using the above linked Google Groups to congregate, cast your vote here. Nov 20, 2013 at 20:34
  • 1
    "Let's all meet in a chat room at 10:00am every 6 or 8 days (that way it's not always the same day of the week, giving everyone a chance to participate), and review close votes together." Nov 20, 2013 at 20:34
  • 7
    Google Group? Nothing whatsoever against Google, but we do have chat, you know... (and given that anyone with enough rep to VTC can participate without much effort...) Nov 20, 2013 at 20:54
  • 3
    What exactly would such a group do? Say "The queue is currently xx.xk lets get reviewing!"? Nov 20, 2013 at 20:56
  • @Dukeling - Admittedly I'm not too familiar with the chat. I mostly wanted to get a headcount at first, and then a way of making announcements like, "We're meeting this Tuesday @ 10:00a. See you there!" Is that possible using the chat? If so, I'm all for it. Nov 20, 2013 at 20:57
  • 1
    @RichardTingle - Yes, exactly. We'd only go forward with this if there were sufficient interest—say, 50 users. Suppose we get even just 20 users to meet per week. If we each reviewed, say, 25 close votes on average (say some users reach the daily limit of 40, and others leave early), then still we'd bring down the queue by 0.5k!—a significant dent in the queue, IMHO. Personally I'd be much, much more enthusiastic about reviewing close-votes if I knew that would be the impact of my hour. Nov 20, 2013 at 21:01
  • I honestly don't think enough people will join said group to make a significant impact - the only people likely to join are those already reviewing a fair amount. Idea - perhaps we can have some sort external contest for reviewing (internal contests have been suggested (Winter Bash... ?)), only for those good enough at reviewing (e.g. no failed audits during the contest, perhaps look at past audits too), perhaps with some prizes - a mod might be able to help here by giving access to some stats not typically available to us common folk. The daily review limit may make this somewhat impractical. Nov 20, 2013 at 21:02
  • 6
    @acheong87 Oh, look, a chat room (not sure what now). Keep in mind that there's >2k new questions in that queue a day, 0.5k isn't all that much (unless this happens daily). Why does going on strike make sense - tending to the close vote queue is like cleaning a friend's dirty house, but it gets messed up faster than you can clean, and the friend's off doing something else, and you listed a bunch of vacuum cleaner brands, with motivation, but the friend can't decide which one to buy. Nov 20, 2013 at 23:10
  • @Dukeling - Great idea w/ the explicit read access! Edited my answer to make it the official roster. And yeah, maybe 0.5k isn't that much—but it's 0.5k more than otherwise. But I think you're right that a strike might be the better long-term solution, especially since a "reviewing club" is external to the system (a band-aid, perhaps); ideally we'd want something internal to the system solve the growing queue. That said though, I like the idea of the reviewing club anyway. I like the idea of logging in one morning with my cohorts and kicking some review queue ass together. Rather than alone. Nov 22, 2013 at 20:13
3

I was going to raise this issue again by holding a Meta Party to "celebrate" the figure reaching 100K. That'll probably happen within the next week at the current rate.

Since you've raised the issue now at 96K, I might not bother with the party, but I hope you'll all have a slice of (virtual) cake on the day anyway.

My thoughts:

  1. Gamification is failing
    What I mean by this is that there is an upper limit of 1000 reviews for a gold badge. After that, there's no further incentive. I'd be willing to bet that a whole lot of people quit reviewing after they've got the steward badge.
    I'm not proposing a solution to this, and yes I agree that if you're reviewing because you want a gold badge then you're doing it for the wrong reason, but... well, if you're going to gamify it in the first place....

  2. Close vote expiry is failing
    When this issue has been raised before, it's been declared a non-issue because old close votes are supposed to expire, and thus the queue ought to just clear itself down given time. This clearly isn't happening.
    I can't remember the exact mechanism for expiring close votes (it was explained, but I couldn't find it), but IIRC votes expire after a given time if the question has received a given amount of traffic but no further close votes.
    It seems to me that the reason that the votes aren't expiring is that most of the questions involved aren't getting any traffic, so the rule above isn't triggered. Which means that our huge backlog is a mountain of stuff that no-body is interested in.
    Again, I'm not sure what (if anything) can be done in response to this, but it leads onto my next point...

  3. Old bad questions are closed by obscurity anyway
    An old question of low quality is virtually certain to disappear without trace anyway; once a question is off the front page, it's only going to get traffic if people are actively searching for it. If it's a bad question, poorly asked and with down-votes, it won't register in searches, so it becomes virtually unfindable unless you remember the exact wording of the question.
    So most of those tens of thousands of questions that have been in the close queue for so many months are not getting any traffic anyway. And given that, closing old questions can seem somewhat futile, which certainly isn't good motivation for reviewers.

1
  • I think gamification is failing at the opposite end of where you think it does, I mean not enough users are able to figure how easy it is to reach 1000 reviews. Of those who get there I believe, most should have learned how to do it fast. To me for example, it took 10-20 minutes to complete daily dose of 40 reviews, an effort comparable to... I dunno, shaving. I don't need a freaking badge to shave
    – gnat
    Nov 27, 2013 at 14:46
3

There's several reasons why the queue grows by leaps and bounds, but never seems to get smaller

  1. You get 50 close votes a day. I'm not sure that needs changing but, in regards to the 1600 miles of beach analogy, if I get only one trash bag I'm not going to be super motivated.
  2. Closures sometimes become moot. User asks a poor/duplicate question and gets a quick answer. Thread more or less becomes moot. Maybe lower the threshold to close an answered thread to three or just remove the close votes altogether?
  3. Not all users are qualified to review all close votes. So a user posts a Java problem and someone says "It's a duplicate of this thread". I'm not a lemming but I'm also not a Java coder and can't tell if there's a difference or not. So either I lemming and vote close or I skip it and hope someone else who can tell the difference comes behind me. This leads in to
  4. Poor tools for filtering closes. I know a ton about PHP but next to nothing about Java. It's a crapshoot what you'll get reviewing that list, tho. So why isn't there a tab for close votes on the PHP tag page? All we have is one monolithic 117k+ closure list. Why not tell me that there's 10k PHP threads awaiting closure and show me the list? You could even make a badge for closures within a given tag in the same way tag badges are handed out now (i.e. 1k close votes within a tag gets a gold badge). If I were in a tag and staring at a much smaller number, I might be inclined to chip in more, especially knowing I'm less likely to run into a situation beyond my expertise.
3

I feel like you:

and have quit b/c my efforts didn't seem to help

This is personal for me too.

I actually asked recently how important it was to shrink the close queue

This was just before I quit doing reviews. I was deeply dis-heartened (just like you!) because I was trying (very hard) to make a difference and I felt like I was using a bucket to empty the Titanic as it sank.

But I found many of the responses enlightening:

The Idea that the close stack (as users clarified it really was), helps to close a lot more new questions and helps set the tone. But in the end, to summarize, there was a lot of ambivalence on the importance and best motivational methods.


I also pointed out recently that here that my audit history was poor and I felt that many users with lower reps would make better reviewers. I was focusing more on suggested edit type reviews. I feel much better about my history of doing close review tasks.

In this post: a recommendation was made to allow users to sort close activities by types by off topic.

I asked to further be able to sort close votes to get just the "Must know minimum about subject", or just "Please do my work for me", or just the "too vague". This kind of granualar filtering would make it easy to run through 40 close reviews quickly (just dummy checking to confirm that the post was really hopeless).


Suggestions for changing a stuck system:

I feel that we could make better use of user's time by letting them filter to get the truly lost cause (obviously case) questions:

This change is particularly important b/c it

  • allows the most to be done with the least effort
  • helps to close questions that are most deserving of closing (again most results)
  • from my experience as many as 75% of questions seem to be of this type.

If we make it easier to review tasks, we could increase the daily allowance of close reviews:

After all, it's easier now!

open up closing reviews to lower ranked users: (Maybe 1K, 2K, not sure)

  • There are geometrically more users at lower rep settings and I believe they can make a big difference.
  • I would hold them to tighter audit requirements and suspend those users more aggressively (but only from closing tasks)

Most revolutionary, all badges for reviews must require a high audit pass rate!

  • this is tricky because users should be able to atone for past wrongs and learn as they go along
  • but badges should not go to users who just sit through suspensions and continue on with questionable practices

Leader boards and badges should be created for users who have high audit pass rates or streaks

  • longest audit pass streak? (per review type?)
  • long current audit pass streak? (per review type?)
  • bronze (10 streak passed audits), silver (30 streak passed audits) , gold (100 streak passed audits)

In these ways I'd like to:

  • increase participation
  • penalize for sloppy behavior (though allow for redemption)
  • reward careful behavior

(Yes I know this post is too long, too aggressive, and suggestions involve a lot of changes that would never happen)

2

When I raised the issue I wrote a reorganisation proposal: Close vote review queue reorganization proposal

While most of it is not true anymore, since the close votes have been refactored, but I think the close votes should still be split up into 5 or 6 different queues, based on the category than being inside one of them, as different categories require (sometimes completely) different ways of thinking, and checking whether the question is actually fine or not.

While this might not stop the piling up of the close votes, it would clearly show how the different types of close reasons pile up, and then another measure might be taken to fix that specific queue.

1
  • 1
    I agree. I've been pushing for filtering, but same idea. Make it easier to select certain types! Feb 15, 2014 at 20:29
1

I have some suggestions and ask for more here:

  1. Review queues only contains the post which have already two close votes (and asked in the last seven days, but older post should be considered with one vote), as one close vote can not always be correct (some users cast vote just because they do not get the post).
  2. Privileges to close the post directly should be at a level where more users can avail it. (Obviously at the level where community can trust them).
  3. Three votes of users with more than 5000 reputation points should be enough to close the post.
  4. A pre-filter apply to the review queue for the users on the basis of tags they are most active in (If I am active user of jQuery give me list of it, so I don't need skip).
  5. Increase the threshold to review the queue from 40 to 60 (or somewhat more suitable)
  6. Highlight the close link for more visibility so more users can cast close vote directly from a post.
  7. Anti-close votes like Leave Open should be on the post itself, gets visible if a close vote is cast, so a wrong misleading vote can be ignored directly from the post without being reviewed in a queue.
  8. One reputation point over 10 reviews (or 10 points over 100 review) will motivate more users to review.
1
  • 1
    I like a lot of your ideas. Just be aware that robo-reviewers exist among 5K users aplenty. But filters, filters, filters... Feb 15, 2014 at 19:26
1

Why not lower the bar for close/reopen votes a bit, so that us lowly 2k'ers can participate?

I know we may not be perceived as equal rights contributors by some of the high-rep folks, and rightfully so, but at this level you do have some fundamental grasp of the community and the quality requirements. There's also a higher level of voters required so it's not likely that this can be abused or fumbled by not-yet-truly-trusted low rep users (at least not something I can think of).

This should also address the point @dmckee raised, about the balance between users adding and removing from the queue.

EDIT: To elaborate a little over what he said - the queue size represents the difference between the people marking questions for close (including flags and low-quality queue reviewers), and the number of people able and willing to review them. Flags are probably not such a large portion of that, but the VLQ poses a lower rep bar so basically you need a single >2k rep user to add something, and 5 >3k users to complete the review. That makes the pipe completely unbalanced.

8
  • 5
    In all fairness us 3k guys don't do a perfect job as it is, lowering the rep requirement is going to mean more low quality reviews Nov 23, 2013 at 13:35
  • 2
    @RichardTingle: the question is, how high do we have to go to get "good" reviews? If the answer is "way too high", just how much lower quality would the reviews become? Nov 24, 2013 at 2:06
  • 1
    @RichardTingle, I strongly believe that after a certain point, rep stops being representative of ability to review and AUDIT PASS RATES say far more. I'd say open up for 2K, maybe even 1K's. Maybe make them pass a test of 50 audit close questions 1st. Definitely audit and suspend them from close tasks easier. But open up and get data on them. They'll suprise you! Feb 15, 2014 at 20:11
  • +1, I've felt this way for a while. and posted such to a lot of snarking. meta.stackexchange.com/questions/208220/… Feb 15, 2014 at 20:12
  • @DaveA, 1K is problematic, you can reach that on a new account within a few days without asking or answering a single post - just by mass robo-editing (which unfortunately has a high rate of approval). 2K is little more demanding. By the way - another way to "solve" this balance is to raise the VLQ bar to 3K, I just don't think that's the right solution.
    – Leeor
    Feb 15, 2014 at 20:37
  • @Leeor, I hear you about ways to get to this point. I counter that a phenomenon of users doing mass suggested edits (to get to 1K) to be able to review questions is such a small and distinct group that it's probly dwarfed by the existing number of 3K+ robo-reviewers. Feb 15, 2014 at 21:31
  • I'd like to open 2K, 1K, not sure users to review tasks, but create other, non-rep hurdles. Maybe a test of 50 audit review close questions. This is an example. Even this test could be gamed, but looking at the big picture, I'd argue that users who care enough to jump through so many hoops and have to pass a test like this will probably tend to be more attentive than the group we hear complaints about. I could be wrong, but we'll never know until we try Feb 15, 2014 at 21:34
  • 1
    @DaveA, With that I totally agree. We should welcome people who wish to help (and can prove they're reliable in other ways than accumulating rep), not block them.
    – Leeor
    Feb 15, 2014 at 21:38
0

At least judging from my (very) short flagging experience,

  1. most flags get handled in a day
  2. some flags take a couple of days
  3. some flags take forever to close

so it's possible people skip close queries that they can't make up their mind about and vote on ones that are obvious.

If this is so, then the reason for the large queue size is because when people skip questions they basically mean "abstain from voting, I'm not sure" and it should be handled somehow as a kind of a "no" vote.

If there's not enough consensus to close a question then perhaps after a timeout period (couple days or so) an automatic resolution for the close vote would be a reasonable measure?

Perhaps something in the lines of:

  • "close the vote because there's not enough consensus for closing the question" OR
  • "if total reputation of users voted so far exceeds a threshold then close the question automatically, else close the vote automatically"

It sure fixes the queue size elegantly :P

Also, lowering the entry point for the close votes system might help, depending on reputation distribution of current active users

4
  • 2
    This sort of happens already. Once sufficient users have viewed a question the votes on it start to "age away". They aren't instantly removed as in your suggestion but they become eligable after a certain time to be removed Feb 11, 2014 at 6:45
  • @RichardTingle, who decides if the votes need to close after they age?
    – bbozo
    Feb 11, 2014 at 8:28
  • 2
    The rules is "Close votes age away harmlessly after 4 days if the threshold is not reached. Each new close vote resets the timer, and close votes only start expiring if the question has had 100 views." See here for more details Feb 11, 2014 at 8:37
  • Ah, @RichardTingle thank you, then perhaps the magic number 100 can be tweaked?
    – bbozo
    Feb 11, 2014 at 10:27
-2

I find this post rather illogical. The queue is increasing in size, yes, but the devs aren't the ones that can fix that. It's the community.

Similarly, it's your site and ultimately, you review so that the site becomes a better place. Going on a strike and telling others to .. that just exacerbates the problem and bounces right back at you.

Yet it seems like SO dev team is doing close to nothing to handle this pressing matter.

I don't know, I don't really think this is a problem the devs can fix.

  • Expiring close votes earlier won't work because the posts aren't getting CV-ignored because of iffyness, they're getting ignored because of the queue size. So stuff doesn't get reviewed.
  • Increasing the number of CVs one can use per day — well, the robo review problems are already large as it is.

Most of the request are along those lines. There are a few promising ones, but some tackle the ill effects of the full queue (eg), and some are aesthetics changes that may improve perception/motivation.

But there's not much the devs can do.


Here's a suggestion though, to those feeling burnt out: Use the filter option on the queue to look at stuff that you feel more comfortable in evaluating. This makes it easy to breeze through the 40 posts. If more people used this, we probably would be better off.

Of course, if you're feeling burned out, taking a break from the queue is the best thing you can do.

10
  • 7
    It would be nice if we could see the size of our filtered queue so we can see how big the problem on our part of the site is; it seems to be largely down to php and javascript, thankfully two tags I have nothing to do with Nov 21, 2013 at 13:47
  • @RichardTingle on the other hand, the "no things to review" will be much more rewarding if you don't. Nov 21, 2013 at 13:49
  • Of course the danger of reviewing on the filtered queue is that too many of the questions are interesting Nov 21, 2013 at 13:50
  • @RichardTingle it's only a danger if you consider closing as much as possible as soon as possible your only goal :-D Nov 21, 2013 at 13:52
  • 2
    @RichardTingle Yes please. Feature request that! Nov 21, 2013 at 13:56
  • 1
    @Manishearth To avoid adding yet annother "O No! Close queue!" post I've added it as a counter proposal here Nov 21, 2013 at 14:11
  • The developers can always change the rules, let say they can increase the number of review per day to 60 or 80 for some population. Increasing the review per day can make the reviewers feel more useful.
    – рüффп
    Nov 22, 2013 at 12:54
  • @ruffp Read the post again, I already addressed that point in the post. Robo reviewers are a big problem and we don't want it to get bigger, the mods already have a tough time battling that. Nov 22, 2013 at 12:56
  • The audit mecanism is not supposed to detect these robots? Aynway if nothing is made to adapt the review process for longer queues, they will always grow without end.
    – рüффп
    Nov 22, 2013 at 13:07
  • @ruffp It's not working fully, mods still have to do a lot of the work. There are improvements to the system being made, but nothing immediate. As for long term growth, I believe the team is already looking into handling that, they just don't have any solid plans. Nov 22, 2013 at 13:21
-2

The following are my suggestions:

Suggestion 1:

Hold mass cleaning day once a year. Some countries in the world who do not have the government officials to do the cleaning for them, have days where all the citizens get outside and start cleaning the streets. All businesses and everything closes and everyone aged 12 to 60 are required to clean the streets.
We can also just have one day where the whole site shuts down and every member visiting must do cleanup. Of course we can add incentive of giving medals and more rep for participation and such. And we will have a requirement that only members who have certain privileges/rep are allowed to clean.

Suggestion 2:

Already has been proposed.

1) Can't we lower the necessary rep to something lower? I am about to reach 1k, but I am very diligent and passionate about this website. If rep can not be lowered at least add active member since requirement. So that anyone who has 3k rep OR has been an active member for 2 year (has 2 yearling medals).

2) Give some incentive by giving more rep for reviewing questions.

3) I believe this is extremely stupid, but seeing how most of the close voted questions are compromised of php and javascript, why do we not get rid of them this one time only? Based on 1 and 2 and fact that there will be more people who will do maintaining this should help things.

-9

How does having a large number of items in the review queue hamper your close vote reviewing experience?

Having a small queue would hamper your close vote reviewing experience much more. If the queue is very small, and often empty, like the other queues, then:

  1. You can't review at your convenience, whenever you want to; you need to wait for there to be items to review
  2. You need to worry much more about race conditions with lots of other people reviewing the same posts as you.
  3. You know that you're not really needed. I used to review suggested edits all the time because the queue was often full. Me reviewing posts meant that more people could suggest good edits. These days, the queue is virtually always empty, so while I could help, my help isn't needed. All the work gets done (and quite quickly) without my help, so I can spend my time elsewhere where that work wouldn't be done without me.

There really aren't really any noticeable disadvantages, from the point of view of a reviewer, to having a queue that is very large.

It is the job of the review queue to draw attention to content that can use the attention of a knowledgeable reviewer so that the content can be improved in some way. When the queues are all sitting at empty it is a failure of the queue to draw attention to problematic content. Having a bunch of empty queues means that a reviewer is forced to not review content using the queues, and are forced to find content to review using their own methods, which is what the queues were designed to prevent. Given that there is almost certainly, given the shear amount of content on the site, going to be content that could use the attention of good reviewers, having an empty queue is simply ignoring that bad content, rather than recognizing that it's there.

35
  • 22
    the queue being empty is a great reward for a reviewer. Close vote reviewers are devoid of that reward. Nov 20, 2013 at 16:01
  • 6
    A large review queue doesn't hamper the process, but it implies that there are many problematic questions that need to be reiveiewed. The fact that the queue just keeps growing and growing (despite repeated attempts by users here on Meta to unsuccessfully address it, even with large bounties on questions) means there is a higher likelihood that really crappy questions just keep sticking around and cluttering the site. I completely get @Shai's frustration that there is seemingly nothing being done to remedy it. Nov 20, 2013 at 16:03
  • @JanDvorak How is it a reward? What is rewarding about it? Someone else did a bunch of work, so you don't have to do it. What is rewarding about that? Review queues are a place to go to find work for you to do, if you want to donate your time. The queues being empty means the site can't find anything for you to do. That is a failure of the review system. To me, a queue that is frequently empty is a queue not being very helpful.
    – Servy
    Nov 20, 2013 at 16:03
  • 6
    @Servy it's primarily a reward if you did put an effort into making the queue empty. I don't want to review. I want to help keep the site clean. If I can review but that does nothing for the site, it's not rewarding. Nov 20, 2013 at 16:04
  • 1
    @LBT So you have to ask yourself what changed between when the queue was shrinking and when the queue is growing. What changed is what items get moved into the queue (in particular, flags put items in that queue, not the mod queue). The queue got better at identifying content that requires the attention of reviewers. The site didn't start generating more bad content the queue just got better at identifying the bad content that was previously not available for review. Having a queue that is better at providing content likely to need attention to a reviewer is a better queue.
    – Servy
    Nov 20, 2013 at 16:05
  • 1
    @JanDvorak The purpose of the queue is to provide you with content likely to need your attention. Due to the nature of the internet, I can assure you that no matter what, there will always be something that could use your attention on this site. (If there isn't, it would mean the site is dying as people aren't coming here to post questions.) Either the queue is doing its job and showing you that content, or it's not, and you're burying your head in the sand thinking that there is nothing for you to do even when there is.
    – Servy
    Nov 20, 2013 at 16:07
  • 6
    @Servy while this is true that getting better at detecting bad questions is good, if we can't close all the questions that need closing, adding more of those to the queue is counter-productive. More questions will get closevotes, but fewer will actually be closed. Nov 20, 2013 at 16:07
  • 6
    That's awesome -- we're now better at identifying poor content, but we can't keep up with the poor content. I used to work in a public library, and I know on the crunch days after a three- or four-day weekend, we would get really backed up trying to process returns. People would get demoralized because our backlog would get so big, but we would throw extra employees at checking the books back in, and eventually catch up. The problem here is that no matter what we throw at it, the backlog just gets bigger, demoralizing a vocal, possibly large user base. We don't have the tools to keep up. Nov 20, 2013 at 16:12
  • 3
    @Servy currently the priority is for questions that got close votes more recently. This means that once a question drops off the front wave, it never gets attention again. If that habitually happens at four close votes, those close votes are lost forever. Nov 20, 2013 at 16:13
  • 1
    @JanDvorak Do you haven evidence that that is indeed what is happening?
    – Servy
    Nov 20, 2013 at 16:14
  • 6
    @Servy uh, no. My primary intention is not to get the queue empty, but to get as many questions as possible rightfully closed. If the queue is growing, some improvements by the dev team probably need to be made in order to get more questions closed. Note I do not support the strike. Nov 20, 2013 at 16:25
  • 7
    Why is having an empty queue "bad"? It means that all the questions that need to be worked on are either closed, on hold, or migrated to another site. It means that things that do need to get migrated can be done so quickly and efficiently rather than waiting for a week or two. It means that things that are already asked as dups can get closed as dups and the OP can get a good answer from an existing one (rather leaving it open for people to restate what has been said elsewhere).
    – user213963
    Nov 20, 2013 at 16:42
  • 1
    There are two kinds of faliure; a failure to detect and a failure to respond correctly/sufficently. The overall effectiveness of the system is determined by whichever is the weakest. Currently the second is the weaker of the two and so it is that element that should be addressed Nov 20, 2013 at 16:55
  • 8
    what a bunch of bull* "Having a small queue would hamper blah blah" - as far as I can tell, only CV queues you have experience with are SO and MSO, there's no way you know how it works when small. I for comparison work in Programmers and Workplace queues (besides SO/MSO) - and per my experience, it's not even close to what you are talking about. 14 downvotes at this "answer" look well justified
    – gnat
    Nov 20, 2013 at 22:31
  • 3
    @Servy you seem to have a lot of opinions on the close vote queue for someone who doesn't seem to use it. I didn't see any votes in your review history either.
    – enderland
    Nov 20, 2013 at 22:50
-9

Good luck with your strike, which I presume is supposed to lead to some negotiating with the counterpart once you have managed to get your message across.

So here is a suggestion for some items to negotiate when you get that far:

Why are 5 (five) members needed to vote to close?

That means that it takes five people with supposedly high(-sih) standards to clean up after one person who posted a quick question, at times without much regard for quality standards.

Doesn't that sound a bit imbalanced?

Also, is there scientific evidence that 5 is the right number? Sounds awfully like a number pulled out of thin air.

Close vote reviewing is tedious, repetitive work, which should be abolished

Luddites aside, modern workers don't want to do tedious, repetitive work, which numerous studies have also shown to be bad for your health, social status, psychological well-being, etc.

We should demand that as much as possible of it get automatised, so we can let the machines do the repetitive work and humans provide the value-adding work.

Removing bad questions is not adding value, it (at best) restores the system to a previous state of presumably higher value. That just amounts to doing a lot of work to get no-where.

If all else fails, just scrap it

It is clear that at its current state, the queue is not working as intended. So if it can't be fixed, drop it. Or just drop the queue counter to 0 (with a new reset every month).

11
  • 10
    6/10 -- Decent trolling, but could be better Nov 20, 2013 at 16:35
  • 1
    If we scrap the CV queue entirely, we'll have even less questions getting reviewed. Nov 20, 2013 at 16:35
  • 2
    automating close votes - how would you automate that? I'm all ears. Dropping the queue automatically? Bad idea. I don't really want to be lied to. Nov 20, 2013 at 16:36
  • 1
    @S.L.Barth scraping the queue will not solve the problem: it will grow back again, because it currently only gets bigger.
    – Shai
    Nov 20, 2013 at 16:36
  • @LBT Why is this trolling? Is there anything factual wrong in the answer or will the proposals not contribute to improve the situation?
    – Monolo
    Nov 20, 2013 at 16:40
  • 1
    Your first point: fair enough. Your second point: How would you propose this works. Your third point: You're right if something is doing a mediocre job it is better to just throw the whole thing away and have nothing instead Nov 20, 2013 at 16:42
  • 5
    To answer your (possibly rhetorical) question, it's 5 because folks complained loudly when it was 1 and kept complaining when it was 3. There's some value in keeping it an odd number, but there's nothing special about 5.
    – Shog9
    Nov 20, 2013 at 16:43
  • @JanDvorak Some sort of pattern recognition could be applied. A closing line of "Any pointers appreciated" is a sign of a question where even the asker is not expecting a clear answer. I am sure there are other patterns that can be identified.
    – Monolo
    Nov 20, 2013 at 16:43
  • 2
    @Shog9 Fair enough - I didn't know that a system with 3 reviewers required had been tested. That said, I am not sure there's a need for an odd number - it is not a vote for or against closing among five people.
    – Monolo
    Nov 20, 2013 at 16:46
  • 2
    @Monolo Majority rules in resolving conflicts (i.e. when some people vote using one close reason and others pick another one). I believe the last vote cast make the decision in cases where ALL close reasons are different, but a majority is a bit cleaner philosophically for the more common case.
    – Adam Lear StaffMod
    Nov 20, 2013 at 21:53
  • 1
    @Monolo Close Vote Reason Logic When There Isn't a Majority
    – user213963
    Nov 20, 2013 at 22:37
-14

If the problem is that a too large queue which is growing is demoralizing how about simple showing 1000+ instead of showing 93125?

6
  • 23
    We might be childish - but not dumb...
    – Shai
    Nov 20, 2013 at 16:24
  • 4
    Almost sure it was already suggested.. Nov 20, 2013 at 16:24
  • 4
    It will soon be true: 1.0M Nov 20, 2013 at 16:27
  • 8
    I don't know, sometimes when I'm actually reviewing the queue briefly goes down. So I can feel like I'm holding back the tide. With 1000+ it's just: You made no difference at all Nov 20, 2013 at 16:43
  • @JanDvorak Its ~90,000 (9 * 10e4) now and soon to be ~100,000 (10e5). Thats an order of magnitude off from ~1,000,000 (10e6). A queue of 1M would mean 1/6th of all the questions on the site (thats not to say that 1/60th of the questions isn't that insignificant either).
    – user213963
    Nov 20, 2013 at 20:07
  • @Michael 1/6th of all the undeleted questions ;]
    – user206222
    Nov 20, 2013 at 21:15

Not the answer you're looking for? Browse other questions tagged .