35

There have been no shortage of ideas to get the close queue manageable. Here's one more: let's fundamentally re-think how many votes it takes to close a question.

Currently, diamond mods can close anything unilaterally, while all other users must always cast a total of 5 votes, without exception. Can we leverage the reputation system—which is, after all, meant to represent how much the community trusts a user—to do better?

20K users are described as "Trusted Users." Can we trust them to, say, unilaterally close questions from the close queue that have negative scores with no answers? Or even negative scores with no upvoted answers? What if we let them, say, cast two close votes for questions (from the queue) that have a score of zero or less?

I don't think I'm alone when I say that I'd be more motivated to tackle that queue if I didn't feel like Sisyphus, more so now that there are over 100K questions in there. It's incredibly de-motivating thinking about how little your daily allotment of 40 votes is doing against this avalanche.

  • 4
    1+ I feel this is the only way to manage the queue. You could even calculate a 'close vote' weight.. or something along those lines. – Josh Crozier Sep 28 '13 at 19:38
  • 1
    I'd also add that the daily 40 limit is too damn low... Just by raising it to 50 could mean a difference... – ppeterka Sep 28 '13 at 19:52
  • @ppeterka - sure. If we did that, gave 20K users more power, and tweaked the queue to show users those questions they have the most power over first (20K users get fed answerless negative questions before any other) then I think we could really start to take the garbage out quickly. – Adam Rackis Sep 28 '13 at 19:55
  • Unilaterally closing things is quite the tool though. I think that might go a bit too far. What I wouldn't mind however is something like a count-for-two vote for them. – Bart Sep 28 '13 at 20:09
  • 1
    @Bart - true. I'm told even moderators have from time to time (with the best of intentions) misused unilateral powers. But surely we could open this feature up in such a way that the chances of deleting good content are slim. A -2 question with no answers that someone else thought worthy of a close vote? Lemme nuke it NOW! – Adam Rackis Sep 28 '13 at 20:12
  • Maybe only within their top tags @AdamRackis? Though admittedly that's not always necessary for the worst of the worst. – Bart Sep 28 '13 at 20:14
  • 2
    @Bart - even with your restriction that could do wonders. – Adam Rackis Sep 28 '13 at 20:18
  • 1
    Related to suggestions 2 and 3 here. – Dukeling Sep 29 '13 at 3:13
  • 1
    I think questions with a low calculated quality score could get closed with less than 5 close votes, esp. since the close reason now more clearly states that the question is "on hold" and can be reopened after improvement. – Gert Arnold Sep 29 '13 at 9:34
  • @ppeterka: 20k users already get 50 close votes a day. Or, at least, I do. I don't think it started at 25k, but I could be wrong. – Wooble Sep 30 '13 at 3:22
  • @Wooble It's not just you, it's everybody. Even I. But it seems you never tried to exhaust them all using the queue, which is impossible, because after 40, it says I should come back tomorrow. I think we should raise that to 50, not the close votes - which is aalready 50... – ppeterka Sep 30 '13 at 6:11
  • Re discussions of linking together: I've checked and this question is already in the index of close vote ideas – Richard Tingle Dec 16 '13 at 20:28
3
+500

20K users are described as "Trusted Users." Can we trust them to, say, unilaterally close questions from the close queue that have negative scores with no answers? Or even negative scores with no upvoted answers? What if we let them, say, cast two close votes for questions (from the queue) that have a score of zero or less?

It seems to me that allowing 20k users to close questions unilaterally may be a bit too extreme, too much potential for mistakes.

Giving trusted users additional vote weight seems reasonable enough though, considering that many of these questions will be auto deleted eventually anyway:

If the question is more than 30 days old, and ...

  • has −1 or lower score
  • has no answers
  • is not locked...

... it will be automatically deleted.

These checks are run every week across all sites.

Allowing trusted users 2 close votes on negatively scored questions with no answers will simply allow these questions to be "shuffled off this mortal coil" a little more quickly, sparing them the slow, agonizing march toward oblivion.

  • 1
    Fair enough, though if we're only talking about casting two votes, I think we can be a bit more liberal and relax the standard to, say, questions scored zero or less with no upvoted answers. – Adam Rackis Dec 23 '13 at 21:08
2

TL;DR

If Stack manually appoint 120 users a month (that's only 4 staff/mods doing one each a day) and lower the total required votes to complete a flag to 3, then close vote queue would be 0 in about 8 months.

Which isn't a bad idea considering it's currently increasing at 10K a month, so that's a total swing of 180K!

If you dislike the idea 3 votes to complete a flag, or 120 users being appointed manually each month, these could be changed if I knew how many votes in total are currently on all active flags.

Also, if you think this is too much, think about other ideas, ponder a bit. We're at 105K now, and in 4 more weeks, 115K, into Feb, it will be 125K...

So while it may not be the best idea in the world, it's one that would work, and something needs to be done sooner rather than later.


What this idea is not!

The proposed idea here is a quick fix to sort the close review queue, not a permanent change moving forward for users with <3K rep to manually be appointed access to close votes on SO.


>3k rep users versus <3k rep users

Users with 3k earned their rep for reasons other than close reviews.
Gaining rep from answering and asking questions, editing, etc, is not an indication they're any good at judging close reviews.

However >3K rep means you are automatically given access to the close votes.
Of course they have likely gained a lot of knowledge of Stack by the time they reach 3k rep, possibly enough to understand close vote requirements, but not necessarily, not every user.

As such, I think there will be plenty of users out there who are <3k rep, who would be just as good at close reviews as those with >3K.


Take me as an example

Here on MSO this week I've just got >3k rep, and so suddenly access to the close and reopen votes.

I've never had any access to these before, so have no experience with them or knowledge of best practice. There's no initiation or training provided, nor is there any decent info on how to action them (other than trailing through search results of people asking and having problems etc).

I can just "have at it" (within reason of course).

How does 3K rep prove I'm worthy to vote on closing and re-opening people's questions?
Ok I've "been around a bit" and so do know a fair bit of the Stack ways, but does my answering a few questions (relating to general ideas and proposals) signify I am able, capable and trustworthy to vote on opening or closing other's questions?

What's to say I'm brilliant at answering questions, but terrible at thinking about requirements for close and open votes?

I believe that a manual evaluation will be better than algorithms.
There are things which a script cannot pick up on, as the script has to be coded by a programmer, whereby parameters and potential scenarios have to be pre-empted.

Also, a script doesn't take into account many things a manual evaluation would, such as:

If your investigations reveal they have:

  • Good accept to deny flag stats
  • Have a good post edit approval to deny rate
  • Decent audit pass percentage
  • Show general diligence and effort to be helpful in questions/comments/answers (just a few looked at, not all)
  • etc

Then surely they are as worthy to attack the close vote queue than someone who has none of the above checked by a script and automatically approved at 3K?
(sure they can be banned from reviewing, but not the point. So could a manually approved user)

When I suggested this a few months back, I think the close vote queue was 80K..? It's now nearly 107K. So while it could be said my idea is long winded and time consuming, while we're sat thinking / waiting of a better solution, the queue is increasing at around 10K a month.
A few hours here and there for staff to appoint some new users to access close votes and they can start helping.

The fact others with > 3K see something is being done they might start helping out more again too.


PROPOSAL

Staff/mods appoint users with >1.5k & <3k rep to have access to close votes.

Maybe put a notice on the community bulletin in the side bar and a link to a question in MSO (and/or blog post etc) telling people who are between 1.5K and 3K to put their name down to save mods having to search.

Although a better approach would be to write a script which gathered some data from the DB's so no-one had to go looking for potential users.
The script returns the data and someone could look at the users and decide if they're worthy etc.

If rep >1500 && <3000 && audit stats = something
etc

So a proposal and some calculations (please bear in mind maths is not my strong point. If I made a mistake, or monumental c0q up, just say..)

If a new manually appointed user actions say 5 close votes a day average, that's about 150 a month per user.

Parameters & calculations used

  • Total additional votes per month per user = 150 (average 5 per day)
  • Total additional votes per month - Multiply (150) by total users per month
  • Total additional flags completed = Divide by 5 (total votes required to complete flag)
  • Current queue growth = 10K per month
  • Stop appointing new users after month 5 (cannot go on forever)

20 users per month per month & 5 votes to complete flag

Month 1 = 3k additional votes = 600 additional flags completed
Month 2 = 6k = 1,200
Month 3 = 9k = 1,800
Month 4 = 12 = 2,400
Month 5 = 15k = 3,000

After 5 months total queue increase = 40K

So the above model is not adequate enough! We need 4 times that just to break even.

So would be unreasonable to find 120 new users a month?

It may sound unlikely, but that's only 4 staff members/mods finding one new user each, every day.
Or using a script to grab data from the DBs then it would be a case of one person checking through the data returned by the script and finding 4 valid users.
Surely can't take more than an hour a day once they've done it a few times?

120 users per month per month & 5 votes to complete flag

Month 1 = 18k additional votes = 3,600 additional closed
Month 2 = 36k = 7,200
Month 3 = 54k = 10,800
Month 4 = 72k = 14,400
Month 5 = 90k = 18,000

Month 4 queue is decreasing around 4K per month
Month 5 queue is decreasing by around 8K a month

Queue total by month 5 = about 100K.
17 months in total for the queue to reach around 0.

If you think this is all too much so far, consider that:
You think it's too much yet the queue after 5 months of doing all this is still at 100K, so it simply signifies the requirement of doing something drastic, and the importance of it being now.

So perhaps consider an additional aid to this:

  • Lower the total users required to complete a flag

If this was dropped to 4, the additional benefit on the model of 120 manually appointed users per month actioning 150 per month each average would be:

120 users per month per month & 4 votes to complete flag
Month 1 = 18k additional votes = 4,500 additional closed
Month 2 = 36k = 9,000
Month 3 = 54k = 13,500
Month 4 = 72k = 18,000
Month 5 = 90k = 22,500

Month 3 queue is decreasing around 3.5K per month
Month 4 queue is decreasing by around 8K a month
Month 5 queue is decreasing by around 12.5K a month

Queue total by month 5 = about 83K.
12 months in total for the queue to reach around 0.

Still not a great model, we're just hitting advantages a month earlier, which stops at 5th month anyway.

If the total users required to complete a flag was dropped to 3:

120 users per month per month & 3 votes to complete flag
Month 1 = 18k additional votes = 6,000 additional closed
Month 2 = 36k = 12,000
Month 3 = 54k = 18,000
Month 4 = 72k = 24,000
Month 5 = 90k = 30,000

Month 2 queue is decreasing around 2K per month
Month 3 queue is decreasing around 8K per month
Month 4 queue is decreasing by around 14K a month
Month 5 queue is decreasing by around 20K a month

Queue total by month 5 = about 65K.
8 to 9 months in total for the queue to reach around 0.

So, they only way this would work is if you appointed 120 new users a month, and lowered the total votes required to complete a flag to 3.

Could be even easier

How many votes are there already on the incomplete flags? If this is substantial then this idea could become much more viable.
Either with less users appointed per month, or total votes for flag completion could remain at 4, or a combination.

I'm sure you have thrashed out many ideas around the meeting table, and obviously none of this takes into account numerous potential parameters.
Such as additional users reaching the 3K rep and helping out, or the fluctuation in other users helping out or not, or the affect of Stack doing this and so the other >3K users see progress and so jump back on to help out, etc.

I don't have access to all necessary data to give an accurate model. This is just an idea.

  • 5
    So you want to also tackle this from the other direction, and find a way to evaluate and let lower rep users into the close queue? We've already had a huge problem with badge-seeking robo-reviews, and I can't help but think this would exacerbate matters. – Adam Rackis Sep 29 '13 at 17:37
  • 2
    I disagree. Rep is meaningless in many ways. Robo reviewers have automatically gained access to closed reviews from their rep level gained from things other than reviewing. If you instead evaluated people based on their current track record of reviewing, their rep level does not matter. I could be a 20k user and robo review, or be rubbish at it. Time on site and rep are not indications of decent users, many just answer/ask popular questions over enough time. Reviewing users manually will likely result in appointing people who are already doing a conscientious and decent job. – James Sep 29 '13 at 20:58
  • Evaluating how good a user is at reviewing is hard. They've set up audits, but the number of false positives is staggering, as the numerous Meta posts show. Moreover, the odds of a 20K trusted user being a robo reviewer may not be zero, but it's far lower than it would be for your average 1–3K user; that's why they're called "trusted users." The idea here is to merely let these trusted users expedite the closure of questions which have a high likelihood of being crap. – Adam Rackis Sep 29 '13 at 21:38
  • Well, I get your point with 20k users, but without facts it's speculation. Them being called trusted users is not because they are trusted, it's simply because one expects them to be given their time and progress on site. Again, rep can be earned in less than trusting and savoury ways. Besides, my point isn't really how trust-able 20k users are, it's that users auto appointed a trusted status regardless of rep level cannot be any better or worse than manually checking someone's answers, questions, review audit and flag stats with a lower rep. – James Sep 30 '13 at 0:11
  • 1
    Are you suggesting that someone with 3k rep (or even 20k) is trusted based solely on their rep total, & so are fit to do close reviews? Is someone at 2.5k not to be trusted then? A users total rep in no way defines their behaviour. My proposed rep level of 1k-2k is nothing to do with some rep based trust activation point, as none exists. Simply that at that point 1k-2k have been here a bit & likely know the site. Besides, there are many scenarios - someone with 50 rep on SO but 3k on Meta likely knows Stack well enough to do close reviews on SO. Rep on its own merit means little. – James Sep 30 '13 at 0:22
  • 5
    I'm not sure you understand how the SO rep system works. Manually vetting users is not even close to scalable for a site the size of SO. Rep is meant to be an automated system of granting these privileges that's right more often than it's wrong; that there are 50-rep users who would be great reviewers is irrelevant. So yes, 20K users are "trusted" and can access a trimmed-down moderator flag queue, vote to delete content, etc. This post merely suggests we additionally trust these users to unilaterally close crap from the close queue. – Adam Rackis Sep 30 '13 at 2:45
  • 1
    Of course I understand the rep system. We're talking about getting the current backlogged close review queue down. So my proposal isn't a universal change, just a quick temp fix. Give a few users some powers they wouldn't otherwise get for a while, but based on real life assessment. Again, rep means nothing. I have 800 rep on SO, does that mean I'm more trust-able than anyone with 500 rep? My proposal is about a quick fix. – James Sep 30 '13 at 2:52
  • 1
    And again, unlocking the gates and letting the masses into the close queue is not feasible. Manually evaluating these users' effectiveness in reviewing is not scalable. Automatically reviewing users has proven hard, and already turns up myriad false positives. And yes, in general someone with 800 rep does not have sufficient experience on the site to know what should be closed and what not. The fact that you may have spent enough time on MSO to know as much is irrelevant. This is about automating a system that's statistically likely to succeed, not cherry picking edge cases. – Adam Rackis Sep 30 '13 at 2:56
  • I think you need to re-read my answer, especially the edit. Who said anything about "letting the masses into the close queue"? And stating "you may have spent enough time on MSO to know as much is irrelevant" is irrelevant in itself. The entire point of my proposal was about manual evaluation of users, not just "open the flood gates" and let anyone with XX rep at it..."Automatically reviewing users has proven hard, and already turns up myriad false positives." so as per my argument manually doing it would be better! Stack doesn't do this because it'd need 1,000 mods doing nothing else... – James Sep 30 '13 at 3:01
  • So you say manually reviewing reviewers would be better, but then acknowledge that to do this SO would need 1,000 mods doing nothing else. Which is clearly impossible. So how precisely do you propose Stack Overflow "evaluate a load of lower rep users" ??? When you have that figured out you should post it as a new feature request. – Adam Rackis Sep 30 '13 at 3:08
  • For Stack to manually review users for privileges is not possible. This is why Stack have the milestones for rep = privilege as an auto function. For Stack to manually appoint some users (say 30) who are not yet at 3k as a one-off so said users can access close vote queue to help out. I've edited my answer. Can see why it was not clear, although my previous comments were clear..Make sense? – James Sep 30 '13 at 3:17
  • 1
    I think the point was not that you would automatically allow people to close who wouldn't otherwise based on rep, nor that you would go through each and every person with 1000-2999 rep and decide whether to allow them the priv based on manual checking of stats. I think the point was, you'd grab a couple hundred randomly, and then manually check only those peoples' stats and select a handful of the best ones. Seems like a reasonable concept to me - I'm a lot better at flagging and reviewing than I am at answering questions quickly. – neminem Dec 16 '13 at 19:56
  • @neminem - that's a reasonable idea, but what stats would you check? Users below 3K don't have access to the close queue. How would you evaluate them other than doing it manually? I suppose you could look at their stats on flagging questions for closure, but boy, I think you'd have a lot more false positives than you would by trusting trusted users. – Adam Rackis Dec 16 '13 at 20:13
  • @AdamRackis You would have to do it manually, that was my point. Why I wouldn't have suggested it even though I'd thought it before, because I'd be laughed at. I still think it's reasonable, though - as James says, you could grab a random subset of people with a sufficiently high number of good flags (and good flag ratio), high enough successful review audit and high enough total reviews, etc., then have mods manually vet some from those into a new category. Would still be work, but it'd be work that'd probably pay for itself quickly. – neminem Dec 16 '13 at 20:25
  • @neminem - well it's not a bad idea, I just don't see the mods wanting to put the time in to manually vet the users (I wouldn't) nor do I see Josh fighting to get this implemented. I'm still not convinced, but feel free to throw it up the flap-pole and see what the community thinks. You won't be laughed at if you state your case intelligently. – Adam Rackis Dec 16 '13 at 20:30
1

NO

I've seen plenty of bad close votes. Having a certain number gives more of a chance for things to get closed rightly, and for the right reasons. Five seems like a good balance.

  • 1
    Are you really worried that too many high-quality, negatively scored questions with no upvoted answers would be wrongly closed? – Adam Rackis Dec 19 '13 at 19:57
  • @AdamRackis, I wouldn't think "high-quality" usually, just more like "normal-quality". And I'm much more worried about bad close reasons in that circumstance than bad closings. – Lance Roberts Dec 19 '13 at 20:20
  • Unless you know there are more good users than bad, then a higher number of users required to complete a flag doesn't dictate the likelihood of it being moderated accurately. There are many robo reviews, many users making bad decisions, so having a higher total of users also brings a chance of getting more bad reviewers. Why is 5 specifically a good balance? You have to consider total good reviewers, bad reviewers, robo reviewers, users who review badly due to bad mood... etc. It's not accurate to simply state more users required to complete a flag = more chance of a correct closure. – James Dec 20 '13 at 0:50
  • @James, I'm definitely working under the assumption that most reviewers are good, and there is only a small percentage of bad reviewers. – Lance Roberts Dec 20 '13 at 0:52
  • @James, I get that assumption from my experience, but I expect before anything like this is implemented that the developers will look at the stats closely to make sure of any assumptions. – Lance Roberts Dec 20 '13 at 1:01
  • Fair enough, if you see that from experience then it's poss true. As said, I've not been in the SO close reviews and I presume MSO close vote area is not a good comparison for me to go by. – James Dec 20 '13 at 1:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .