10

The close vote queue currently holds 62k+ posts and has been rising for months now. The dropping numbers at the beginning are long history.

What can we do to reduce the queue? One possibility would be looking at the rules that keep questions in that queue. Normally close votes expire within 4 days, but not if the question has fewer than 100 views.

Why do we have that restriction? Nobody seems to view the question, so let the close votes expire.

I propose to remove that 100-views restriction and let close votes expire after 4 days. This Query shows that close votes on 54k* questions should expire then.

Or allow closure with 3 votes having less than 100 views in 4 days. But that reduces the queue only of 10k questions.

* Did I miss something in that query? I am surprised that the number is so high.

16
  • 3
    An alternative would be to switch privilege levels on cast close and reopen votes and approve tag wiki edits, allowing more people to go through the queue.
    – Ryan Frame
    Jul 24, 2013 at 13:51
  • 5
    IMO it might be helpful to filter by the number of votes questions have. Then if we wanted to get through questions with 4 votes, etc, those could be handled immediately.
    – Taryn
    Jul 24, 2013 at 13:58
  • 15
    Let's fake the number and put 120 or something small so people will want to clear it. When you see 62K you don't feel like! :) Jul 24, 2013 at 14:02
  • 8
    Voiding the votes will only clear the queue...what's with the close-worthy questions? Jul 24, 2013 at 14:05
  • @bluefeet: Just sort them by votes descending...that way at least those with four votes get pushed out of the queue very fast. Jul 24, 2013 at 14:06
  • @M.NightDemonbobby On the 10k tools that is great, but from the review queue it would be nice to have the ability to filter by number of votes.
    – Taryn
    Jul 24, 2013 at 14:13
  • @RyanFrame Perhaps I'm misunderstanding, but wouldn't that make it worse?
    – Mansfield
    Jul 24, 2013 at 14:59
  • Why does the close vote review queue need to become smaller? What harm is being done at its current size?
    – joran
    Jul 24, 2013 at 15:01
  • 1
    @Mansfield: I guess it probably would. I was thinking more potential closers means more questions get closed.
    – Ryan Frame
    Jul 24, 2013 at 15:18
  • @RyanFrame I don't disagree with the logic - but you'd want to lower the reputation required for the review queue to achieve it.
    – Mansfield
    Jul 24, 2013 at 15:19
  • 2
    @Mansfield: Oops. I meant tag synonyms rather than wiki edits (lowering from 3000 to 2500 was my thinking)
    – Ryan Frame
    Jul 24, 2013 at 16:05
  • @RyanFrame Ahh okay, that makes sense. But then I'd think you'd have to lower the threshold for casting close votes as well which may be less than ideal.
    – Mansfield
    Jul 24, 2013 at 16:41
  • What exactly is non-ideal with lowering the threshold for the privilege casting close votes? Reputation does not exactly correlate with the ability to make correct moderation-related judgment. Jul 24, 2013 at 21:54
  • @Old not exactly, but it's the best gauge that we have (except possibly meta rep, but I don't know how people feel about that).
    – Cody Gray
    Jul 26, 2013 at 1:01
  • @CodyGray, I'd say a combination of % helpful flags, number of helpful flags, and reputation (with a lower threshold, say 2000 or even lower) would be a better criteria for awarding close vote privilege. But that would require modifying the reputation-based privilege system, which seems radical (likely to elicit eeeks) Jul 26, 2013 at 1:09

8 Answers 8

27

Please don't do this. Crap questions don't get less crappy because they have low views.

Please implement one of the auto-closure suggestions.

4
  • 5
    Does it do any good to have them sitting the the review queue forever though?
    – Joe W
    Jul 24, 2013 at 19:18
  • @JoeW: It's still better to not have them in queue, in my opinion. Jul 25, 2013 at 8:26
  • @Rosinante, do you have links to any of these "auto-closure suggestions"? I like the sound of it, but can't recall having seen such a proposal.
    – jscs
    Jul 25, 2013 at 18:41
  • "Crap questions don't get less crappy because they have low views"...But I'm betting "Crap questions get low views because they are on hold but not closed"
    – Harrison
    Oct 16, 2013 at 19:10
22

This isn't a terrible idea, but - as others have noted - that 100-view minimum exists for a reason: it was very difficult to close questions in low-traffic tags without moderator intervention without it.

Besides that, it misses the real problem here: the close review queue backlog is growing because the number of questions entering it require more reviews to complete than are actually occurring on a daily basis.

Here are some hard numbers:

  • 1388 review tasks created in the close review queue per day over the past 30 days
  • 2583 reviews performed per-day in the close review queue over the past 30 days
  • 1230 users reviewing items in the close review queue during the past 30 days
  • 15166 users with close privileges active on Stack Overflow during the past 30 days

In theory, it only takes one review to resolve each review task - if you edit a post, that drops it out of the queue immediately. In practice, many questions can't be fixed by editing, and many more would require more effort than reviewers are willing or able to put in... So the real number is 3-5 reviews per review task. With the current number of active reviewers and a limit of 40 reviews per day, we're nowhere near enough active reviewers to keep this queue from growing. Some of the moderators pitch in and clear out a large number of items (mod responses are binding and unlimited), but there's no way they can handle the entire surplus.

So what can we do to fix this? Well, there are several options, and we'll probably want to try all of them if possible:

  1. Reduce the number of visible items in the queue. This is purely psychological - folks get discouraged when they put in a lot of work and the big number just gets bigger. If they don't see their work having any effect, then they stop doing it.

    We don't need to actually throw away the votes, but there's not a big reason to actively review questions with year-old (or even 2 month-old) votes. If these ever get another vote, they'll be back in the queue and fairly close to the top - otherwise, they can drop out of the queue.

  2. Encourage more people to review. For suggested edits and flags, we display a number at the top of the screen - a sort of "bat signal" for reviewers. As we transition to review queues, flags are becoming less and less important for 10K users - so, we could just as well replace that with a close-review indicator.

  3. Encourage more specialized reviewing. It is much faster to review questions for closure when you filter the queue to just the tags you're familiar with and focus on specific close types. But... Right now, a lot of folks don't realize that this is even possible. Making the filter UI more obvious, perhaps displaying a tag-specific "bat signal" on the tag pages for top tags or splitting duplicate closing out into its own queue would go a long way toward encouraging folks to be more effective reviewers.

Finally, it's worth noting that because of how the review system is designed, the backlog of review tasks tend to have no reviews at all, as previously-reviewed items rise in the queue and are completed. That's good - nothing would ever get done otherwise - but it also means we can't do anything clever like auto-closing posts after 3 "trusted" close-reviews or completing reviews after a single "trusted" do-not-close response. That's unfortunate, but fixing it without breaking the ability of the system to accomplish anything will be a bit trickier.

5
  • Just curious: any numbers on what percentage of close-reviews indeed end up in closing a question? (In other words: "how much agreement" once a question gets into the close queue?)
    – Arjan
    Jul 24, 2013 at 22:30
  • Roughly 76% of questions reviewed in the close-queue end up closed when the review is completed.
    – Shog9
    Jul 24, 2013 at 22:37
  • 1
    Agree with "Reduce the number of visible items in the queue". I do tend to think "why bother?" sometimes even though I know I'm helping Jul 25, 2013 at 7:58
  • Highlighting the ability to filter on tags would be good. I've done around 725 close reviews and, whilst aware of the filter on closure types, have managed to miss filtering on tags.
    – borrible
    Jul 31, 2013 at 10:55
  • "the backlog of review tasks tend to have no reviews at all, as previously-reviewed items rise in the queue and are completed" How about picking one or two off the bottom and promoting them to the top each day?
    – TRiG
    Nov 20, 2013 at 18:07
14

I would like to see the queue present questions with the most close votes first. Voting to close a question with four close votes has a more immediate impact than voting to close a question with a single vote. Additionally, the questions with more close votes tend to be more obvious closure candidates; this allows reviewers to move through the queue quickly.

Eventually we will blow through the "easy" closures and be left with the harder ones, but we receive the immediate benefit of a cleaner site by closing as many bad questions as possible, as quickly as possible.

I just realized that M. Night Demonbobby proposed the same thing via comments quite a while ago. So, props to him.

7
  • More close votes first would just mean that busier tags would get yet more traffic (ie, a tag with enough people to get a question to 4 votes on its own will get those questions reviewed quickly and closed, even though in the scheme of things that tag likely have a fifth vote on its own) while a tag with few followers will have lots of questions sitting at 1-2 votes that never get reviewed.
    – Joe
    Jul 24, 2013 at 14:53
  • 2
    @Joe That's not necessarily a bad thing. High-traffic tags need to be kept as lean as possible. Otherwise, they get overwhelmed by the noise. Tags with lower traffic don't suffer as much because the fewer number of total questions tends to make avoiding the bad ones easier. Additionally (in my experience) the high-traffic tags appear to attract a larger percentage of bad questions because of the low barrier to entry. Jul 24, 2013 at 14:57
  • 2
    That logic doesn't make any sense. High traffic tags a) get lots of valid questions and b) have lots of people voting to close - how long does it take to close a question on c++? 5 minutes? Low traffic tags, on the other hand, get easily overwhelmed by bad questions because none of them get closed unless they're on the review queue.
    – Joe
    Jul 24, 2013 at 15:06
  • For example, this question was opened and closed 3 minutes later. No question in sas would be closed within that length of time - it would be days, if ever, without the review queue.
    – Joe
    Jul 24, 2013 at 15:12
  • Aren't the questions with the most close votes already candidates to be closed without the need for the review queue? Using this sort order might even only get to the low hanging fruit, which would haven fallen without the review queue anyhow? (I'm not sure if @Joe is trying to say the same thing.)
    – Arjan
    Jul 24, 2013 at 21:54
  • Actually, that was proposed by someone else in another question about the queue...but I can't find it right now... Jul 25, 2013 at 8:33
  • The 10k Tools pages for Close and Delete Votes both have "Most Votes" lists, so there doesn't seem to be any reason not to do this -- as one of the filter options -- in Review as well.
    – jscs
    Jul 25, 2013 at 18:39
9

This might be a short-term solution, but the 100 views minimum is a good thing for tags that don't get very many viewers (in particular, tags with relatively few who have close vote privilege). Obviously bad questions, like this one, deserve to be closed; if the queue is temporarily too large to get them to several reviewers in a few days, they'll lose the few close votes they have and never get to be closed.

Perhaps the compromise is to have a longer limit - say 10 days - for <100 view questions. If the close queue is > 10 days long, small tags are probably being hurt more than they're being helped anyway; and this would help them out by having a higher proportion of 'small' tag questions in the queue.

4

Indeed:

Close votes (and probably reopen votes) only expire after all of these conditions are met:

  1. 4 days have passed since the last close vote.

  2. The question has more than 100 views.

These 100 views include those of users who cannot even vote. And unless "100 view requirement for vote expiry should not be based on total views" is actually completed, the view counter is not reset after a vote is cast.

Now, on one or two occasions, I have run into a question I wanted to close, but could not as I already voted to close the very same post some time in the past, which expired (so: more than 100 views). That makes me wonder: if I happen to run into the same question twice, how many others might meanwhile also have voted to close, and had their vote expire too?

So: are we sure that expiring this quickly doesn't mean that the very same question keeps getting into the review queue for 4 days, each time only having one or two votes cast by different people, expiring before reviewers see it?

My gut tells me that the expiration should be increased (delayed), to not need a review queue for closing. So:

I propose to remove that 100-views restricion and let close votes expire after 4 days.

Instead, I'd say: increase those numbers.

And I guess this can easily be played with by just changing some configuration?

(But I have no figures to back that gut feeling. And I'm not a reviewer.)

1

The close vote queue is a goldmine of information, so let's embrace it

While I tend to agree with those who suggest that it should just be flushed, I also have to say that this discussion is going in circles, and a fresh approach to the queue is needed.

A recap

The ideal meaning of a close vote is that a community member has analysed a question, found a specific problem with it and taken the trouble to vote to close it.

This is also how close votes work for many (most?) questions, where people see the question in the queue, do their own "analysis" and agree with the original voter and add their vote to the tally. Unless, of course, they disagree and vote to save the question from the queue.

However, the questions in question stay in the queue because they have been around for a few days without four good men and women having added their vote to the close process. After that time, they are so far down the queue that practically nobody will see them again in the context of a close vote review. They will linger there forever, unless they somehow reach 100 views so their accumulated votes start to decay.

What does this mean?

Given how the queue works, the questions they contain have in reality been reviewed by the community (or a small part of it, but that's how the queues work) and deemed to be... meh.

Or another way to put it: the questions in the close vote queue have been weighed and found to be not bad enough to waste time and a close vote on, but not good enough to save, either.

Now, this is some real information about the quality of the question.

The quality assessment information provided by the community

If we accept this source of information, we have the following quality information for ranking purposes:

  • Up- or downvotes
  • Lingering close votes
  • Actually closed questions
  • Deleted questions

There may be a few more that I have missed, so please feel free to suggest them in the comments.

And before you dismiss this analysis totally, please notice that I am aware that "quality" is a loaded word, and in the above I have used it in the broadest sense. A closed question is not inherently a bad one (it may for instance simply be off-topic,) but relevance is also a quality, so I don't see a problem in using the information for ranking purposes.

It should also be considered that the close-voting process has a certain tediousness to it, as it takes at least 3 clicks to vote to close a question, as opposed to a single click for a downvote. Therefore, a close vote could easily be considered on a par with a downvote in the quality rankings, maybe even more.

With this approach, the much criticised close vote queue changes from being a source of frustration (i.e., a liability) to become a source of information in the quality ranking of the posts on the site, and hence a real asset.

1

What if we faked the number and said there was like 1000 questions to be reviewed? Maybe even just remove the number? That might make people less discouraged from doing it. We could even raise the 40 post restriction.

6
  • 1
    But are people discouraged from doing reviews because of the high number? Jul 24, 2013 at 21:52
  • @OldCheckmark I can say I am...
    – Cole Tobin
    Jul 25, 2013 at 1:06
  • If it's going to be fake, why not just get rid of it?
    – Ryan Frame
    Jul 25, 2013 at 4:02
  • @RyanFrame If we say there are 0, people might think it's empty...
    – Cole Tobin
    Jul 25, 2013 at 4:06
  • 3
    But why have a number at all? What's the point of me seeing 1000 when I know it isn't right?
    – Ryan Frame
    Jul 25, 2013 at 4:08
  • @RyanFrame oh. I thought you meant say 0. Removing the number might be a good idea.
    – Cole Tobin
    Jul 25, 2013 at 7:05
-2

An alternative approach to reduce the queue size, might be to disable casting the first close vote on a question, until the user has reviewed N (e.g. 10) questions from the close votes queue. This would achieve the following:

  • It would increase the number of reviews, since you would have to review some questions before each first close vote.
  • It would reduce the number of questions entering the queue, since some people would give up instead of reviewing questions.
5
  • 4
    If people give up and quit casting close votes entirely that doesn't help maintain the quality of the site's content.
    – Ryan Frame
    Jul 25, 2013 at 4:01
  • No not entirely. But the people that add items to the queue, should also help reduce its size. Otherwise, it will just keep growing forever.
    – user000001
    Jul 25, 2013 at 4:40
  • 1
    @user000001 It's like saying that if you want to take broken glass bottle from a sidewalk and put it into street trash bin, you should also participate in trash disposal process. Sure, if people would be willing to do that it would be neat, but requiring it would most often than not simply cause broken glass to remain on the street untill someone gets hurt. Not a perfect analogy, I know, but illustrates how "preventing trash bins from overloading" should not be the goal on it's own.
    – Mołot
    Jul 31, 2013 at 11:09
  • @molot I see your point... Perhaps I popup reminding people to review items after they cast close votes, like the one that reminds to add a comment when downvoting, would help,without compromising the quality of the site.
    – user000001
    Jul 31, 2013 at 13:12
  • With popup reminder I can agree all right. If it would include a link to review queue filtered by current question tags it would be even better.
    – Mołot
    Jul 31, 2013 at 13:16

Not the answer you're looking for? Browse other questions tagged .