31

Check the Review stats for Stack Overflow. Low quality posts is effectively empty, after previously having thousands and thousands of items. The suggested edits queue is kept low, after previously remaining near-stuck at several hundred. But the close votes queue count is still sky high with no sign of shrinking.

An innate challenge of the close vote queue is it requires more votes; 5 votes to close (4 once it's in the queue) or 5? "do not close" votes to remove the post from the queue and start close-vote aging. More reviews are needed per item to clear it from the view; only 2 votes on suggested edits (or three with a tie-breaker) and (I think) just a couple looks good votes on a Low Quality post remove it from the queue (or three delete votes, or 6 recommend deletion votes).

So what can be done to efficiently and effectively (accurately) process these posts? Is anything slowing you down/keeping you out of the close vote queue? How can this process be improved?

Improve this question
12
  • 12
    There are two things that make reviewing questions a lot slower for me. 1) If it's not in my direct area, it takes me a lot longer to judge it - if at all. 2) The questions are mostly "meh" type questions. They're bad questions, but I don't feel strongly enough to purge them.
    – Mysticial
    Commented Sep 1, 2012 at 18:59
  • 3
    If we could start filtering that could push things right along
    – random
    Commented Sep 1, 2012 at 19:07
  • @Mysticial I notice 2) a lot, fleshed that out into an answer. RE: Random, I think it tries to show you stuff you might be interested in, not sure how smart it is. Probably not very.
    – Zelda
    Commented Sep 1, 2012 at 19:09
  • 2
    Also, 95% of the stuff in the close-vote queue could be technically closed as "Too Localized". But there are plenty of people willing to answer those.
    – Mysticial
    Commented Sep 1, 2012 at 19:15
  • @Mysticial or duplicates...but that's an SO problem, not a review task problem :)
    – Zelda
    Commented Sep 1, 2012 at 19:16
  • How about we add a mod only "Close everything with at least one pending close vote" button to it? And it doesn't really have to be mod only...
    – yannis
    Commented Sep 2, 2012 at 0:13
  • 3
    @YannisRizos only if we implement a feature to funnel all ensuing drama and rage directly to you
    – Zelda
    Commented Sep 2, 2012 at 2:31
  • 2
    @BenBrocka Bring it.
    – yannis
    Commented Sep 2, 2012 at 2:36
  • You're assuming that the suggested edit and low quality queues have been processed effectively and accurately :-).
    – ben is uǝq backwards
    Commented Sep 2, 2012 at 12:19
  • 1
    There are still 141,401 low quality posts.
    – fuxia
    Commented Sep 8, 2012 at 4:22
  • It has started to shrink actually, at about a rate of 150 per day ... so next year we'll be done :)
    – Jack
    Commented Oct 2, 2012 at 7:27
  • related: meta.stackexchange.com/questions/151833/…
    – gnat
    Commented Nov 14, 2012 at 22:24

12 Answers 12

Reset to default
18

Two things I'd suggest to change on Close Vote Review Queue (CVRQ).

First, the question thread should be fully visible in CVRQ detail layout, I often go to original question to find out if the question is answered, how it is answered, if and how are others rating the question and it's answers.
Maybe it seems inappropriate, but I feel that question is determined by community attention, and cannot be reviewed just from final score of question, number of answers and if the OP has an accepted answer.

Second, It happens to me that I do see question in CVRQ that are marked for some close reason, but occassionally there are no close-votes, it is probably marked by Community<> robot, but anyway, it should be visible. I'll then go to questions detail, and I see no close-votes on question, and I'm truly confused

Update 22.11.2012

In CVRQ I'd expect to see the number of votes to close directly without need of clicking button Close, I have to do this in ~90% of cases. The number of close votes currently raised on given question is in my opinion important decision criterion.

Improve this answer
1
  • 2
    I don't agree that the number of close votes is a important criterion. If it was we could decrease the limit instead.
    – Emil Vikström
    Commented Nov 22, 2012 at 19:25
14

An idea I had based on Mystical's comment; age close votes after a certain (large) number of Not Sure "votes".

Why? Because close vote aging is supposed to happen when lots of people look at a post and don't feel strongly enough to close. If a question gets 20 "meh"/"not sure" reviews, IMO that's as good a reason to start aging close votes as 100 views (the current requirement) because those are 20 people who could have voted. (The number is a totally arbitrary suggestion.)

The threshold should be higher than Do Not Close, but I notice I cast a lot of "Not Sure" in the close vote list. If a lot of people review the post and no one is really sure what to do, maybe that's a sign it's just not close-worthy.

However I also think this needs to coincide with more strictly showing people close votes on questions in their interest areas, or only allow "aging" from Not Sure "votes" from people who have earned rep in the question's tags. I'm thinking a lot of people might just think "Javascript? I don't know that mess. not sure"

I don't want to just arbitrarily knock stuff out of the queue, but this thick grey area seems to be a huge problem, and at some point all these grey area posts make it harder to review the actual these need closing/not closing posts. Knocking out the grey area posts allows the queue to dwindle faster and people can instead focus on new posts that clearly need close, not old posts no one has opinions on.

Could any devs share some stats on how common "Not Sure" is as a review choice in Close votes, and maybe some breakdown of how many "not sure"s posts that go unclosed tend to accumulate over time?

Improve this answer
5
  • 5
    We would still need a "skip" button, though. I use the "Not sure" option for that, if I don't feel I know enough about the subject to weigh in, and if Not sure gets the meaning you suggest, we'd still need a way to just skip a question we don't want to have an opinion about.
    – Monolo
    Commented Sep 1, 2012 at 19:12
  • 4
    @Monolo hence I'd like some weighting or a better way to get [tag] posts seen by [tag] contributors. If 20 C# answerers Not Sure a C# post IMO it's quite likely not worthy of a close. But people like me will probably click Not Sure because we have no idea about C#--so why are you even showing me C#?
    – Zelda
    Commented Sep 1, 2012 at 19:15
  • exactly. But even within one's "own" tags, you might want to just skip a question without having to analyse if it is worth keeping needs editing (== work) or deleting.
    – Monolo
    Commented Sep 1, 2012 at 19:23
  • FWIW, the queue is currently set up to show you questions from tags you've participated in (similar to the unanswered page's "My Tags" criteria) and/or included in your "favorites". It's not a sure way to achieve relevance, but it's a good start. If you don't understand the topic, or really aren't sure if a question should be closed or not, "Not Sure" is a way to move on without influencing the outcome - but if you simply aren't convinced a question should be closed, click "Do Not Close".
    – Shog9
    Commented Sep 8, 2012 at 4:27
  • I skip a lot of questions because my area of expertise is related to webdesign and front-end development, not drivers in C or GIS. So +1 to @Monolo and BenBrocka
    – FelipeAls
    Commented Sep 9, 2012 at 19:55
12

I think a good idea is enabling the browsing by tag. I find it easier to judge posts on topics I am used to. If I review close votes and after 20 posts I clicked Not Sure I eventually get tired and quit.

I believe that if I can browse by tag this will not occur as I will quickly realize if the post deserves to be closed or not. More over, I may later edit it to improve it if I voted not to close.

I think this option will encourage people to review.

Improve this answer
1
  • 1
    This exists: that's largely the purpose of the "filter" link next to the close vote review queue's title. You can select up to three tags, and the queue will be filtered to contain only questions in those tags.
    – Mac
    Commented Nov 11, 2012 at 4:37
11

I can only talk from my own experience, but when hanging out on SO for a day I burn right through my close votes (even without using /review).

You have no more close votes today; come back in 21 hours.

This prevents me from doing any work on the review system.

P.S. this might just be the dreadful tags I'm hanging out in.

Improve this answer
7
  • 2
    I honestly hadn't considered this, wonder how big of a factor it is. Probably pretty major on SO, on smaller sites I never run out of close votes.
    – Zelda
    Commented Sep 2, 2012 at 2:30
  • Again: I don't know whether it is the fact that I'm following just tags with terrible questions in it. But for me it's kinda hard to handle the daily "crap" pouring in and using the review system. :)
    – PeeHaa
    Commented Sep 2, 2012 at 2:38
  • 2
    The backlog is a big problem; ideally the review system is there to help handle the daily crap efficiently as possible. But now there's 4 years of crap in here, and unlike low quality it hasn't (that I've noticed) shrunk
    – Zelda
    Commented Sep 2, 2012 at 2:46
  • Perhaps there has to be some organized team effort to clean it up. In php chat we use a system where we can post a question for others to review. In this way people can focus on 1 question at a time and make sure it gets the 5 votes needed instead of people just randomly voting on 1 question which just gets on the pile to never get anymore close votes. However I don't know whether this will work for 50k+ questions. Also considering the voting cap.
    – PeeHaa
    Commented Sep 2, 2012 at 2:51
  • Good point. Would it be feasible to increase the vote allocation for closes from the queue only, and (if you feel it's too risky) only for posts of a certain age? Four-year-old posts may not warrnat the same cautions as newly-posted questions.
    – Monica Cellio
    Commented Sep 6, 2012 at 15:10
  • 2
    @MonicaCellio That's what I consider requesting. I've suggested to remove the vote cap entirely for items in the queue, but the community did not really agree with that. So I am thinking of making a new [feature-request] / [discussion] to at least make more possible votes possible. I like your idea about making a difference between old / new posts. Imo it is clear that something needs to be done. because I see the queue only gettng larger instead of smaller.
    – PeeHaa
    Commented Sep 6, 2012 at 15:14
  • @PeeHaa, another effect of restricting it to the queue is that you can't use the extra votes to initiate close action (i.e. being the first closer). I'm only on smaller sites where none of this is an issue, so I don't know whether that (and the age thing) would address the concerns on SO of just raising the cap wholesale.
    – Monica Cellio
    Commented Sep 6, 2012 at 15:17
7
+50

I suggest putting only questions with four close votes in the close vote queue, at least at first. When those questions are exhausted, questions with three close votes can be added, then two, then one.

In general, the greater the number of close votes, the greater the consensus that a question should be closed.

I'm also thinking there should be more filters. For example:

  • Filtering questions with a score of X or less.
  • Filtering questions asked X days ago or sooner.
  • Filtering questions asked X days ago or earlier.

Here is another idea. I suggest splitting the "Close Votes" queue into several smaller queues, classifying questions with close votes by their age. For instance, there could be a queue for questions asked within the last six months, questions asked 6 months to 1 year ago, and questions asked earlier than 1 year ago, or some other reasonable grouping.

Improve this answer
2
  • 2
    The bad thing about this is that you cannot vote "Do Not Close" until it hits the review system. Your suggestion makes for an unfair advantage to the people pouring over certain tags to hunt close-worthy questions (I'm not saying that those people are necessarily wrong or evil, just that they shouldn't be the only ones to decide).
    – Emil Vikström
    Commented Nov 22, 2012 at 19:28
  • Because of that I made another suggestion that doesn't rely on the number of close votes as much.
    – Peter O.
    Commented Nov 23, 2012 at 0:53
7

A certain threshold of Do Not Close votes should remove a question from the review queue (and queue aging of at least one close vote?). An example of a question that has lingered for a month in Server Fault's queue:

https://serverfault.com/review/close/12627

Jeff Ferland reviewed this just now: Do Not Close
Ladadadada reviewed this 6 hours ago: Close
Michael Hampton reviewed this Sep 3 at 8:32: Do Not Close
Ward reviewed this Sep 1 at 18:33: Do Not Close
rnxrx reviewed this Aug 29 at 1:07: Do Not Close
ewwhite reviewed this Aug 8 at 16:16: Do Not Close

When do people stop seeing it for review?

Improve this answer
2
  • 2
    It does happen. IIRC the close votes start aging after 5 “do not close” votes, which may be too much.
    – Gilles 'SO- stop being evil'
    Commented Sep 6, 2012 at 18:00
  • This has been changed to 3 Do Not Close votes (or one mod do not close)
    – Zelda
    Commented Nov 10, 2012 at 16:27
3

There may be some useful additional filters that would help by reducing the time needed to study each post by prequalifying them in some way.

For example, I'm very fast at reviewing "Too localized" questions that have an accepted answer. The OP is obviously satisfied, so I don't worry about that - I just have to quickly check whether it might be useful to anyone else. If I could filter for "answered too-localized Java questions" I'd blow through them pretty quick.

I suspect other people might have similar filters that would help them.

Much less significantly... I'm sure there are many folks here who will tell me I'm doing the wrong thing (so don't bother!), but I often look at the answers in deciding whether to close a post. It would be a bit faster if I could vote from the fully-opened post.

Improve this answer
1
  • +1 for vote from the full post. It's good that I can vote to close from the question, but sometimes I see a post with a close vote I disagree with, and I can only vote to close. Surely it's sensible to let me vote to keep open rather than hope to find it somewhere in the review queue. This would sometimes help remove grey area questions from the queue.
    – AndrewC
    Commented Nov 10, 2012 at 0:36
2

The queue should stop showing the same "not sure" questions multiple times (at least not on the same day). Maybe it's because I access /review from multiple computers/devices? Not sure if "not sure" votes are stored in the database (I hope so).

Improve this answer
1

I believe that the close-vote system is seriously flawed. Sometimes questions are badly formulated but do still have a valid core of general interest (especialy questions from beginners for beginners). I tried to answer a few of them but the question although still very young was already closed during the time i was answering. My attempt to let the moderators reopen a question so that it can be better formulated failed.

To address your question:

How can the Close Vote review queue be improved?

I would like to suggest the following. If some suggestions are already the case then please forgive my ignorance:

  • First of all i suggest that each question should be x hours old before anyone can vote that the question should be closed
  • give the people that have already answered the question a chance to either improve the question or a veto-right against closing the question
  • warn users and the users that answered about a pending close
  • if a user who has answered the question vetos against the closing it should take more people to close the question
  • if a user who has given an answer and is willing to improve the question let him do it . meaning that the review process should take into consideration that the question will be closed unless the question will be massively improved
  • do not hinder users to improve a question after the first closing votes.
Improve this answer
5
  • 1
    Your logic is a little flawed. The point of closing is to give users a chance to fix their posts without worrying about invalidating answers. This process in no way hinders users from improving a question, at all. This is why closed posts are still editable, but not answerable. Also, what you're proposing makes it harder to close, which won't help reduce the size of the queue.
    – jmort253
    Commented Nov 11, 2012 at 4:12
  • If a question is closed can it be edited even by new users with little points? Here i suggested to edit the question and the user replied he could not do so or here my edit suggestion failed although the user agreed to improve it. May be the review process is flawed.
    – slackmuggle
    Commented Nov 11, 2012 at 18:34
  • It looks like you gave the user a link to the suggested edit queue, which that person won't be able to view. I'm not exactly sure if the person would have been able to click the edit link and approve the edit or not, but the person can definitely make his or her own edits. Also, moderators didn't close a question, those are regular users with > 3000 reputation. Moderators have diamonds by their names, and only 1 moderator is required to close a question.
    – jmort253
    Commented Nov 11, 2012 at 19:29
  • I donät think we should have a time limit before making it possible to close questions. Such a limit would (probably) drastically increase the number of duplicates and bad questions on SO since a lot of bad duplicates will have time to accumulate good answers, thereby giving the asker less incentives to refrain from posting duplicates. Duplicates and really bad questions should be handled/moderated as early as possible in my opinion.
    – Emil Vikström
    Commented Nov 22, 2012 at 19:31
  • If it is a exact duplicate than it should be closed asap - but in this case i would suggest that the question is no longer visible at all. Closed unanswered questions that are a duplicate are just noise.
    – slackmuggle
    Commented Nov 23, 2012 at 22:31
1

We should not only filter on a user-chosen set of tags, but increase a user's vote weight based on his/her reputation on answers by tag. For example, if a user has over 1000 rep from answering questions and that user says a java tagged question should be closed, then it's a pretty good be the question should be closed. There's probably little to be gained by waiting for four other reviewers to vote to close that question.

We can debate the formulas, but we need to acknowledge and exploit the fact that some users have a lot knowledge related to some tags while not being that much of a useful engine on other tags.

Improve this answer
1

More advanced interface

Give us a nice interface allowing searching, sorting, etc. similar to what you can do with questions.

I'm not too fond of the black-box approach to doing things.

Imagine trying to answer questions in the same way - here's a question, can you answer it?

Leave the black-box approach behind or allow users to choose.

Users who don't review often may feel more comfortable with this proposed interface.

Allow your tags preferences

I know you can currently filter on 3 tags, but why not allow all your tags preferences to be in effect (and possibly allow this option to be turned on/off at the user's request)?

No matter which 3 tags I pick, there will always be a reasonably large portion of questions which I know nothing about, thus I must skip these questions unless it's completely clear what to vote. My ignored tags would mostly take care of this.

EDIT: Allowing all my tag preferences would probably allow me to pace through the questions (with accurate voting), where as now I'm digging through them. May motivate users to review more.

Improve this answer
-2

I would like to suggest that 'exact duplicate' and 'off topic' should need only 3 votes instead of 5 to trigger. (If all 3 are to the same duplicate or alternate site.)

Unlike the others, both of those are not (or are much less) subjective, so shouldn't really need lots of votes.

This should reduce the number of items in the review queue (although I would be curious to know by exactly how much).

Improve this answer
6
  • 1
    I'm getting downvotes. Could someone do me the courtesy of saying why? My suggestion would reduce the review queue - isn't that the goal?
    – Ariel
    Commented Sep 6, 2012 at 8:32
  • 2
    On meta, downvotes mean disagreement.
    – Toon Krijthe
    Commented Sep 6, 2012 at 9:06
  • 1
    Migration is a feature that (on SO) has been misused to a significant extent (ask the Programmers mods :P) so generally anything that makes that easier is likely to meet contention
    – Zelda
    Commented Sep 6, 2012 at 13:09
  • 2
    For exact duplicates this makea a great deal of sense.
    – Monica Cellio
    Commented Sep 6, 2012 at 15:11
  • 4
    I strongly disagree about exact duplicates. If you read carefully these should be relatively easy to judge, but I still see a ton of inappropriate duplicates in the queue. They look similar at first, but the differences are clear on closer reading. Sometimes these questions are getting close-voted even when there's a comment from the questioner pointing out the important differences. Don't make this problem worse. [CC @MonicaCellio]
    – Jeremy
    Commented Sep 9, 2012 at 4:34
  • 1
    I'll defer to @TheElementofMagic since I don't know the situation on SO. I haven't seen that pattern on the sites I'm active on, but they're all much smaller so maybe the problem is less likely to arise. On Biblical Hermeneutics, Mi Yodeya, Writers, and The Workplace, three close-duplicate votes would generally mean it really is a duplicate.
    – Monica Cellio
    Commented Sep 9, 2012 at 15:06

Not the answer you're looking for? Browse other questions tagged .