76

When we were working on the last , I wanted to introduce something into the review system that I still feel is pretty lacking. There's no way to really track your efficacy as a reviewer.

For instance, how many of your close votes actually resulted in something being closed? How many posts that you worked on in the helper queue actually went on to do rather well? In other words, how many times did the action you prescribed actually turn out to be the best action to take?

It didn't go in because showing these kind of stats to folks is difficult; it's hard to present them in a way that isn't critical when you're trying to be encouraging. It was also a lot more work than I anticipated.

Anyway, my point - we've been talking about the idea of scaling close votes based on the amount of rep you have. I don't think that's a bad idea fundamentally, but I have some concerns:

  • Really good reviewers also often turn to reviewing, moderating and janitorial stuff as their primary way of participating. It's great if they have gobs of rep when that happens, but not everyone does. I barely coasted to 20k on Stack Overflow just from 'interest' on my old answers once I became more interested in the human aspects of the site.

  • It would give a lot of people more votes, and I don't know if that's what we really need. What we really need is for the folks that have honed their review and moderation skills more votes to work with.

I'd like to open a pretty broad discussion just to get input and ideas. I really feel like we should be giving people more votes based on how accurately they tend to cast them, but I'm also not opposed to giving some additional votes as people reach notable rep levels.

So let's say both ideas are open, would you award 5 extra close votes after 15 questions the user voted on were ultimately closed? More / less? Should 10k users get an automatic additional 10 votes provided that they've voted to close at least 10 questions successfully?

It's really easy to get kind of complicated here, so I'm open to broad ideas. We're going to be starting the second iteration of the quality project in the not too distant future, and I'm hoping to nail down at least the first 10 things that will be in scope (and this is one of them).

  • 3
    it's certainly better than scaling based on close experience but probably worse than based on tag badges. When one does things organically - in familiar tag - everything is so much easier and feels so much more rewarding "...try review filtered by duplicates in an unfamiliar tag... just be careful and drop it when your brain will start boiling." – gnat Jul 29 '16 at 16:51
  • 4
    @gnat you just taught me to read the instructions first. I'm going to need an absurd amount of ice. – Tim Post Jul 29 '16 at 16:53
  • 3
    (and wait until Will figures out that I wrote the original title specifically to annoy him) – Tim Post Jul 29 '16 at 16:54
  • Overall, I'd say yes. If possible it'd be great to also award them for the number of consecutive passed audits you've had. E.g. at each 1k rep interval and 10 passed audits, you get an extra CV. People who review enough to need more close votes review regularly enough that the infrequency of audits will not be a factor, IMO. – TylerH Jul 29 '16 at 17:18
  • I wonder what the stats are on what percentage of close voters with good close-voting records max out on a regular basis. (I primarily use smaller sites, so I'm just trying to get a feel for how big of a problem this is network-wide.) – HDE 226868 Jul 29 '16 at 18:06
  • 1
    I think it's probably an edge case but I wonder how often active users VTC early and the question gets edited and fixed so that it does not need to be closed... that sort of thing probably shouldn't count against a user... though, again, that's probably an edge case. – Catija Jul 29 '16 at 18:11
  • 2
    @TylerH remember that most sites don't have Audits, so using audits to determine CVs has limited usability. – Catija Jul 29 '16 at 18:11
  • 3
    @HDE226868 max out stats may be somewhat misleading as there are users who force themselves to stop closing after they have 1-2 votes left, just in case if something terribly awful pops up (example). More accurate stats would likely include users who reach max-2 to max votes – gnat Jul 29 '16 at 18:25
  • 2
    You can check your own review action against that of your peers with this user script I wrote. – rene Jul 29 '16 at 19:28
  • @gnat max-2 might not work on smaller sites. I cast a lot of CV on SF and almost never get close to using all I have available simply because theer aren't enough close worthy questions. – user147520 Jul 29 '16 at 19:28
  • 3
    I can only speak for myself, but ever since I got to 3k, most of my time on this site is spent shoveling figurative garbage, and besides casting a bunch of close votes in the SOCVR channel, I also have over 1,2k completed close vote reviews. This would bring me next to no use, as me getting to 20k would happen around the time hell freezes over. – Magisch Jul 29 '16 at 20:55
  • 1
    @TimPost Awesome! Gonna go drop that comment in Charcoal HQ. – ArtOfCode Jul 31 '16 at 18:01
  • 3
    @ArtOfCode We talked about poking at it before, and then a bunch of stuff kept coming up, which is why I'm making it a big part of this project. I'm pretty excited too. – Tim Post Jul 31 '16 at 18:08
  • 4
    Just to clarify, is this primarily aimed at Stack Overflow, or also at sites that consistently manage to keep their closevotes queue under 100k outstanding reviews? – E.P. Aug 2 '16 at 19:56
  • 1
    Can we expect some official feedback on this or are we just burning rep. for nothing here...? cc @TimPost – kayess Feb 1 '17 at 9:35

12 Answers 12

39
+150

This answer is entirely from the angle of a primarily dedicated close voter on SO. My version of that, myself, is one in which only a few questions seem intellectually intriguing per day to want to answer. So it works out well. Mostly because I can earmark the question until later, that being a question not subject to FGITW anyway.

We have the SO Close Vote Reviewers chat room that performs community moderation in a larger group of peers reviewing questions. The SOBotics chat room for various experimental bots searching content. And the Campaigns room for one-off occassional drives to close the close-worthy for whatever tag someone desires (hoping they provide the reviewers too). Naturally, there are other rooms. Petter from SOBotics has a bot posting links to other rooms quite focused on their thing.

In Campaigns a few days ago we close to tripled the closes in one day for c#. Bots do a lot of filtering prior to our eyes and brains needing much work to spot things. Mainly by downvote and close vote counts. As such, we are able to participate across many tags each day even if they are not in the tags of our immediate expertise or comfort zone. Frankly, most of those questions are just begging to be closed.

The problem is, we run out of votes. Note also that much of this is performed outside of the close vote review queue. Some of us work off of lists and do not have the mental anguish of "is this an audit?" I can't recall the last time I was in a queue.

In an ideal world we would not be called upon to do this in such small numbers of actual participants. Close voting, afterall, doesn't pay much rep now does it. Yet, there is nothing keeping us from sanely reviewing up to 75 or more per day on the days we are so willing. Like in Campaigns, but not everyday. Close voting does tax the brain.

So what do I suggest? Allow for vastly expanded close voting privileges for those pro close voters that have this interest, skill, track record. There is no guarantee we will use it everyday or even 30% of the time. We do try to find some balance in our lives and pursue other things, afterall.

  • 4
    "Allow for vastly expanded close voting privileges for those pro close voters that have this interest, skill, track record." I'm really supporting that idea. – πάντα ῥεῖ Oct 13 '16 at 23:28
34
+200

If the goal is consistent, high quality content? Then yes, by all means make it easier to close and ultimately delete/remove low quality stuff.

My recommendation is to analyze the huge quantities of existing data SE has to identify what the characteristics of good closer voters are. Some suggestions:

  • How many users have 90% of the questions they vote to close remain closed without an edit
    • Filter out questions that go close vote -> edit -> reopen/not close to not get counted against users
    • What are characteristics identifying "good close voters" and "bad close voters"
  • Are there higher thresholds than 90% you can identify?
  • What are reputation profiles for people who actively participate in closing, editing, and review queues? Does higher reputation correlate?

It would be much better to use more intelligent thresholds for something like close votes than "merely" reputation. Maybe make a "close weight" score. Every successful "valid close" adds 1. Every invalid close subtracts X. I don't know exact details but something like that.

I also think that the circles for "high rep" and "active reviewer/closer" don't overlap much. So if the goal is equipping people actually doing the closing/reviewing to better do so, make sure that reputation is actually a meaningful lever.

The bright side is if someone figures out the above bullet points they will know the answer to those questions, too.

And last, an important hypothetical question, is someone with 30,000 reputation and 0 close votes better equipped to close questions than someone with 5000 reputation, 5000 close votes cast, with 3000 reviews completed?

  • 1
    To your last point, that's exactly what concerns me - "here's a ton of ammo! By the way, check out our guns." – Tim Post Jul 29 '16 at 17:57
  • @Tim fortunately you have tons of data to mine ;-) – enderland Jul 29 '16 at 18:00
  • 3
    Also I'm pretty sure a similar system already exists for flagging (better flag history --> more flags) but I'm not 100% sure. – enderland Jul 29 '16 at 18:10
  • 2
    @enderland system for flags is described here: "You start with 10 flags per day. Based on your reputation and flagging history, you can end up with as many as 100 per day..." – gnat Jul 29 '16 at 18:28
26

Like others, I'd tend to deemphasize reputation as a factor in this. Using the percentage of close votes that result in closures that stay closed is a good idea, as enderland suggests. Here's one more idea:

Factor in the ratio of close votes vs. leave open votes on questions I vote to close.

Why is this a big deal? Consider these scenarios:

  1. I cast the initial vote to close on Stack Overflow, and no one else votes before my vote expires, so the question stays open
  2. I cast the initial vote to close on ELU, and three users after me vote "leave open."
  3. I cast the initial vote to close on a beta site, three users after me vote to close, and no one selects "leave open," but my vote expires before the question is closed.

In (1), you don't know much about the accuracy of my close vote. Maybe it was appropriate, but maybe not. In case (2), it's pretty clear that my close vote was probably a bad one. In case (3), on the other hand, it's pretty clear that my close vote was appropriate.

Thus I'd suggest emphasizing the ratio of close votes to leave open votes. A metric based on closed questions that stay closed may be useful in conjunction, but I think the ratio I've described is likely to be more indicative of appropriate close voting.

  • I think most questions that are really close-worthy won't have the close vote expire. Isn't that the point of the vote expiration? But I could see not counting scenario (1) in either direction. I guess that's what you're driving at. – Wildcard Nov 18 '16 at 6:43
  • 1
    @Wildcard Right, (1) doesn't give us much information so I wouldn't count it either way. As for (3), I wouldn't say it's common, but I've certainly observed it in communities where the vast majority of close votes are cast by one of 7–10 users, and not all of them vote daily or even weekly. So yes, "most close-worthy questions" won't have expiring close votes, but I think it's non-trivial enough to account for it. – Nathaniel is protesting Nov 18 '16 at 16:54
22

I agree with many of the answers above: people who have high rep usually aren't the people who are doing moderation. For instance, on my main site, Arqade, I find myself doing lots of reviewing and editing, but not much answering. Hence, I have hovered around 4k rep for quite some time.


But, the main thing I want to say is: if you do base the number of close votes on previous accuracy, make sure you know the right formula to do so:

equation for correct sorting

See the linked article for more details, and implementations in Ruby and SQL. (Dislamer: I have nothing to do with this blog or article.)

Note that the article is talking about positive and negative ratings; you can simply replace positive ratings with useful close votes, and negative ratings with incorrect close votes to achieve the same result.

What's wrong with a more (simple) formula?

There are two formulas which are commonly used (some of which have been already suggested in this Q), but are not the best formula for the job:

  1. Score = (Useful CVs) - (Incorrect CVs) - works well if all users have cast the same number of CVs, but if someone has cast a large amount more than the others at a higher error rate, they may still be considered a 'good CVer'.

    Example: User 1 casts 600 CVs. One third are incorrect. Score: 400 - 200 = 200. User 2 casts 100 CVs, all are correct. Score: 100 - 0 = 100. According to the formula, User 1 is better, but we know that is not the case.

  2. Score = Average rating = (Useful CVs) / (Total CVs) - works well if all users have cast many CVs, but gives a deceptive result if a user has not cast very many CVs.

    Example: User 1 casts 500 CVs, only 10 incorrect. Score: 490 / 500 = 0.98. User 2 casts 5 CVs, however none were incorrect. Score: 5 / 5 = 1. According to the formula, User 2 is better, but in reality, User 1 is probably better, because they have more experience.

Why this formula is better

The above mentioned formula incorporates both the proportion of good-bad CVs, and the uncertainty caused by a low total number of CVs, which means that it covers the weaknesses of both of the above mentioned examples. Yes, you could simulate this by adding a second or third requirement (e.g. you must have cast at least x close votes at y% accuracy within z amount of time), but the above formula (should) be easy to implement with little tweaking.

  • 1
    I agree with the point you're making, but this answer would be a lot more helpful if it (briefly) explained the algorithm rather than just quoting a formula and giving a link. – David Z Aug 2 '16 at 8:30
  • 1
    @DavidZ I have edited the post to include the advantages and disadvantages of various formulas, and why this is the best. – angussidney Aug 2 '16 at 10:15
  • That certainly helps, thanks. – David Z Aug 2 '16 at 10:47
  • 1
    But ONLY use information from the close review queue, so that close voting rings don't confirm each other, and hence increase each members close power. – Ian Ringrose Oct 19 '16 at 17:44
  • @IanRingrose, I think that's an excellent point, and one that I was concerned about reading through the answers here. It has an extra advantage of driving more users to the review queue. However, it has an associated can of worms—what about a question where three close votes came from the viewing page and two from the review queue? – Wildcard Nov 18 '16 at 6:45
  • @Wildcard, I expect there will be enough questions where at least 3 votes comes from the review queue to generate the stats that area needed. – Ian Ringrose Nov 18 '16 at 10:01
  • It's a Bayesian formula :) – theMayer Nov 28 '17 at 19:38
17
+150

Most people don't use all their close votes, and on many sites you couldn't responsibly use 20+ close votes in a day anyway, unless site scope just changed and a bunch of cleanup is needed. And as others have noted, high rep isn't a good predictor for reviewing activity. So I think we want some sort of adaptive system, not a blanket "ok, make that 30 (or 40, or 50) instead of 20". We should instead look for ways to give effective reviewers more votes, as other answers have noted.

Definition: a disputed close vote is one where the question was not closed (reviewers disagreed or vote aged away) or where the question was closed but then reopened without edits.

My proposal is in the spirit of our other rolling limits. If, in the last $time_period, somebody used all his daily close votes at least $days times and had fewer than $disputed% disputed, increase his limit by $extra_votes. If $disputed% ever reaches $too_many_problems, reset to the default number of close votes.

The exact values should be tuned by data analysis, but maybe something like this: If, in the last 30 days, somebody used all his daily close votes at least once and had fewer than 10% disputed, increase his limit by 5. If the (overall? recent?) proportion of disputed close votes reaches 25%, reset to the starting number of close votes.

With a scheme like this, an effective close-voter gradually gains more votes the way an effective flagger gradually gains more flags.

As I said, the numbers should be tuned. But the idea is that so long as somebody is (a) voting well and (b) bumping up against the limit, gradually raise the limit. If those votes aren't being used well, reset. (Possibly also start aging away the extras if they aren't being used. I haven't thought through that scenario.)

  • 2
    bounty awarded for a laser sharp focus on an issue that worries me most and on approach to its solution – gnat Aug 7 '16 at 20:23
11
+100

To expand upon the current answers.

I do not think giving more close votes alone is the optimal way to go. I think close votes should become weighted with either rep or tag badge progress (although tag progress can be slow).

Agreeing that reviewer fatigue is an issue as pointed out in this answer with these results.

To answer I am proposing an additional option. This algorithm would require two factors:

Part A

So to use the type of algorithm you are proposing on increasing number of close votes I would recommend that when a user reaches 10K rep that the close vote counts as two close votes.

So a question would require:

  • 5 close votes from (user <10K)
  • 3 close votes from (user <10k) + 1 close vote from (user >10k)
  • 1 close votes from (user <10k) + 2 close vote from (user >10k)
  • 3 close vote from (user >10k)

This could be tiered to increase the close votes for users > 20K to count as 3 close votes.

  • 2 close votes from (user <10k) + 1 close vote from (user >20k)
  • 1 close votes from (user >10k) + 1 close vote from (user >20k)
  • 2 close vote from (user >20k)

So close votes would carry (to coin a phrase) a weighted count or close vote count.

To close a question requires a total close vote weight count of 5.

Close vote counts:

  • A question requires 5 close vote counts to close (except dupe hammering).
  • A user with rep >3K and <10k has a close vote count of 1.
  • A user with rep >10K and <20k has a close vote count of 2.
  • A user with rep >20k has a close vote count of 3.

Part B

As not all users are equally active in moderation, there needs to be a check to ensure quality of close vote weights.

So the above alogrithm needs to be inclusive of the following algorithm.

This should remedy the close review queues more quickly than increasing the number of daily votes.

For each 1K of rep the voter must have minimum 100 close votes cast that have resulted in the question being closed, let's call it a benchmark of close vote success.

Benchmark of close vote success:

  • Close vote weight of 1 does not require a minimum benchmark of close vote success.
  • Close vote weight of 2 requires a minimum benchmark of close vote success = 1000.
  • Close vote weight of 3 requires a minimum benchmark of close vote success = 2000.

By not including a percentage of successfully closed questions over total close votes, we are not penalising people for reviewing in less popular tags of discouraging people from reviewing at all. We don't need to do that.

It does, however encourage users to vote to close questions that already have close votes rather than voting new questions into the queue, if they are concerned about the close vote success benchmarks.

  • This becomes problematic when coupled with close vote reasons, particularly migrations and duplicates. How you would you deal with that? – Catija Aug 4 '16 at 16:30
  • @Catija same way, as a majority split, based on vote count = vote. If a user's vote is trusted to counts as two, then the close vote reason also can be counted as two votes with that reason. – Nobody Aug 4 '16 at 16:49
  • I'm sorry but I strongly believe that one person should not be able to migrate a question on their own (excluding mods, of course). – Catija Aug 4 '16 at 17:03
  • @Catija well it could be tweaked to reflect that, this is an idea,a mere seed, and if the community likes it, then the community should mould how it takes shape. – Nobody Aug 4 '16 at 17:07
8

As elaborated in some other answers, few people use up their close votes on a regular basis and thus giving more close votes to those users who already have some is unlikely to help. I already argued in my answer on Proposal to make close votes scale with rep that the logical step would be to grant the close privilege to users who do not have it already.

While my answer to that question was based on reputation, there are other reasonable criteria to award close votes to users below 3 k reputation:

  • Reviews in other queues, in particular reviews of questions in first posts (as those pertain to the task of a close reviewer). For example, for each 20 questions you reviewed in first posts, you gain a close vote. To ensure that you reviewed carefully, one could deny this close vote if:

    • you failed review audits;
    • you cast too many upvotes or no action needed on posts that were later closed (or, even worse, deleted as spam/rude/abusive).
  • Successful close flags, i.e., helpful close flags. (IIRC it’s a subsequent close vote that makes such a flag successful, which makes sense as it accounts for edited questions.) To ensure a minimum familiarity with the site, it seems reasonable to put some reputation threshold on this, e.g., 500 (the reputation needed to review first posts).

As an alternative to keep things simple and avoid confusion, one could simply count each flag for closure like a close vote if the flagger meets certain criteria similar to the ones listed above. The only disadvantage is that some low-reputation users with a high number of flags (e.g., from flagging spam or comments) can cast a lot of close votes this way. However, should they abuse this privilege, they also cast unhelpful flags and there is already a system in place to block them.

Finally, as an icing on the cake, it also makes sense to apply similar mechanisms for users above 3 k. E.g., if you performed a lot of first-post reviews on questions, you gain some bonus close votes.

7

I hear those on the "Overall rep != Quality Reviewer" side, but the simple facts are

  1. You can't just shotgun rep (except Jon Skeet). There's a cap per day so you have to earn it over time. There's a reason 20k users are uncommon. Is there a chance someone might just "coast" to 20K? Sure. But that would be something done over years. And if you're not actively contributing over that time and hit 20k anyways it's not likely you'll be interested in moderating anything.
  2. We tie moderation to this anyways. The best example here is the gold badge Mjölnir. One shot close powers for duplicates. If we can't trust a bare minimum 10k user with that kind of power, whom can we trust?
  3. It still takes 5 votes to close anything else. And Closures aren't permanent (hence the Reopen queue). So on the off chance people get things wrong, other people can get it right.
  4. You get more flags the more flags you have accepted. You always get 50 close votes no matter if you have 3k and started yesterday or 300k and 40k close votes (now that's dedication!)

If we can't trust someone who has hung around here for years and has hundreds of upvotes, who can we trust?

The flipside?

  1. SO gets ball-parkish 6k questions a day. A lot of them are in terrible shape, asking things that have been asked before or don't deserve an answer.
  2. Professional Low-Quality Reputational Workers will often troll these questions in hopes of some quick reputation. So the questions get closed, but not quick enough to make a difference.
  3. The answer gets an upvote and thus the Roomba is out. So now you need to get 2 other (10k+) people to come back and vote to delete it. Or you just let it hang out there.
  4. Reputation is gained if you just leave it.

I did a search the other day for a common PHP problem. I found some 2400 questions that appeared to be asking the same thing with nearly 1000 that were closed. I can't help but think that some of those might be deleted now if it they had been closed quicker.

So I say more close votes for more reputation.

  • Somewhere shog posted a statistic of how many people actually use their daily close votes in entirety, and its surprisingly few. I'd be interested in finding out how many more close votes that would exactly enable. – Magisch Jul 29 '16 at 20:57
  • 1
    @Magisch That actually strengthens the case to do it then. Minimal risk – Machavity Jul 29 '16 at 21:08
  • 5
    The point of retiring the term "RW" was to avoid labeling people. Not to begin a euphemism treadmill. – Laurel Jul 31 '16 at 0:27
  • -1 for what @Laurel refers to. You're stating an opinion and masquerading it as a fact. Stop. – Wildcard Nov 18 '16 at 6:53
7

I don't actually think the number of close votes people have is a problem. The idea of blanket more close votes has been attempted, and deemed a failure (together with review queue limit increases) before.

Shog9 said in his answer here that very few people even regularly hit the cap, so I don't think just giving them more votes will make an appreciable difference.

TL;DR: I would like to see some data on wether or not higher rep users are actually reliably hitting the cap before thinking that it would be necessary to give them more.

That said, reviewing 40 close votes a day is ... draining. Not many people do it, and the ones who do don't do it every day. Unless you're organizing via chat, I can barely see many people naturally hitting the close vote cap, unless they're on SO all day.

The matter of fact is, the core problem on SE sites is that closing bad questions is a purely altruistic task, while answering them instead usually gives you a bunch of rep. Its the reason why people concerned with quality often flatline around 3-4k rep. If you have the close votes and care about quality, its quite hard to find stuff to answer at reasonable speed.

So long as the direct (reputation) incentive is actively discouraging you from closing a terrible question over answering it, closing questions will always be an uphill battle: You'll have to fight the askers who don't like their question closed and want answers and you'll have to fight people who want quick points for answering the 19500th dupe question "re: What is a Nullpointerexception in java?".

  • Your answer is a bit disappointing because it doesn't address the question. I read the question as: let's get more votes on, you only come-up with don't do that. I assume you do agree with a quality improvement of the posts on SE, starting with SO, If scaling votes isn't the way forward, what are other options? – rene Jul 29 '16 at 21:17
  • 5
    @rene Its a X-Y problem. The idea behind the question is if giving high rep users more votes will have a positive effect / make the situation better. The answer to that is no. Maybe I just read it wrong, but before we discuss whom to give more votes, I think its worth looking at wether more votes are actually part of the solution. – Magisch Jul 29 '16 at 21:19
  • @Magisch There was a MSO featured experiment (last year?) that had proven that more votes didn't have a significant effect. – πάντα ῥεῖ Oct 13 '16 at 23:32
  • @rene I'd also support scaling votes actually. – πάντα ῥεῖ Oct 13 '16 at 23:33
5

(Estimating reliability of the voter is addressed in separate section at the bottom. The rest of this answer assumes that there is a way to have it.)

What we really need is for the folks that have honed their review and moderation skills more votes to work with.

If you want to leverage skills of reliable close voters and can somehow estimate these consider "teaching" the Community ♦ user to cast votes to close. I mean, make it cast a binding vote on a question that has enough close votes (3 or 4) from voters who are sufficiently reliable.

Clear benefit of this approach is that it is strictly focused on what you want and doesn't involve risky changes that could lead to hard to control side effects. You experimented with 3CV at Programmers and with 60 reviews at SO and every time you were getting undesirable side effects, and you don't really know if any other indirect way to leverage voters experience turns good enough.

Directly "converting" voters experience into Community closes has minimal side effects and you can easily tune and tweak the criteria that triggers it, for example exclude duplicates, questions with many views, positive score etc. If needed you can set specific per-site criteria or even completely turn it off.

One difficulty here could be how to break ties on close reasons. For a start, Community could abstain of casting a vote when there is no clearly winning reason. Maybe if you choose to account for "reliability score" of the voters when resolving ties this will become a corner case.

Another thing to take care of is to ensure that Community votes do not count when you estimate reliability of the voter. This is to avoid an effect of self-fulfilling prophecy. "Community cast a vote because I am known as reliable voter and this additionally increased my reliability estimate" - do that 100, 1000, 10000 times and your estimate may seriously skew.


Speaking of suggestion that triggered this discussion, to scale close votes based on the amount of rep, I would want to share my understanding of what people wanted in this request (and many similar ones). It is way too often that I see this interpreted as people willing more questions to be closed (and subsequently translated into concerns of how to handle more closures).

The way I see it, it's really different. People who complain just want to hit close limit less (or get too close to it). They don't really want more closures, they hardly care about this.

This may sound like a subtle difference but I think it is important. If my understanding is correct then the way to address the issues raised in requests like this is to figure category of users experiencing this kind of issue and find out how system can (if it can) grant them extra close votes with minimal impact and minimal risk.

And when I say minimal impact, I mean just that. Don't try to leverage extra votes obtained this way (at least until you gain sufficient evidence that these go the way you want) - instead try to make sure that these make minimal impact. For example, if you expect or observe significant increase in closures, you can try hedging this somehow, say by making votes expire faster. Focus solely on making users happier without breaking the system while doing this.

First thing you need if you pick this way is to decide what kind of users experience the issue. Does one who bumps or gets too close to the limit 100 days in a year qualify? Does one who bumps once a week? once a month? This is important because the better you estimate them, the better are your chances to get rid of complaints with minimal impact of the change.

Next thing is to pick category to best cover the group of users you want to help. Are they mostly active reviewers, or tag badge holders, or high-rep users etc? Picking the right category and narrowing your target users is important, again, to minimize the impact of the change.

The last but not the least is to estimate how reliable are close voters in the category you picked. This is critical because this defines how you will handle the issue.

If they are generally reliable, you grant them extra votes. If they aren't... well, you decline the request.

Sorry, we would be happy to give you extra votes but our stats indicate that most users experiencing this problem tend to vote wrong. We have strong reasons to expect that giving extra votes will lead to increase of incorrectly closed questions.


How to estimate close voter reliability

I believe that there would better be a dedicated, separate discussion focused solely on that. There are just too many topics to be addressed to properly fit it here.

For example,

  • Is it safe to only use votes to close for simplicity or other related actions need to be taken into account - reopen, Leave Closed, Leave Open?

  • Is it safe to ignore deleted questions for simplicity? (I bet it isn't but put it here explicitly to make sure it isn't lost)

  • Is it safe to ignore expired / retracted close votes for simplicity?

  • Is it safe to ignore edited / reopened questions for simplicity? (at SO it probably is but what about smaller sites)

  • Is it safe to ignore close reasons for simplicity? Is a voter who cast 1000 "unclear" votes on 1000 questions that were closed as "too broad" reliable?

  • Is it OK to ignore question score for simplicity?

  • How to account for review audits? What about disputed audits - those that become ineligible due to human factor correction?

  • Should "reliability score" be displayed publicly, or only to the user, or kept internal? What about "concerns that it would encourage problematic behavior"? (fate of the accept rate suggests that these are better taken into account)

  • 1
    Using Community is another route to my suggestion of weighting close votes. Isn't it easier to weight the votes up front and then everyone knows where they stand, rather than some algorithm and then the post is mysteriously closed? I imagine the flood of meta posts about it. – Nobody Aug 2 '16 at 12:16
  • I also don't think more close votes is the answer (although I once did request it) after the trial, it's exhausting reviewing so many close votes. I would like to close vote and have more confidence that the question will be closed. That is why I've not been bothering of late, The queue feels so huge and the task too hard. – Nobody Aug 2 '16 at 12:26
  • @Yvette showing "reliability score" publicly may have negative side effects, see note at the bottom of my answer. Also, knowing current "weight" won't help those looking at questions closed in the past. How could current weight of voters help figure what happened in a question that was closed a month... a year ago? Wrt more close votes, it is certainly the answer for many of those who regularly hit the limit, I myself feel fine when this happens to me but I have seen folks at Programmers who were apparently very frustrated, I want the site to become more comfortable for them... – gnat Aug 2 '16 at 12:30
  • ...not at the expense of getting more incorrect closures though :) – gnat Aug 2 '16 at 12:30
  • @Yvette one thing you pointed to is really troublesome in my approach: Community closes indeed may look mysterious and suspect. I myself have decided to suggest it only because I saw how experiments to do this without it have failed. I think if SE decides to give it a try (one good thing is it can be easily turned off if we see that it works bad) then they will need to pay a very close attention to estimating reliability of human voters involved. Only this can make Community closes work... – gnat Aug 2 '16 at 14:01
  • ...it can work only if its closes are based on reliable votes. People should look at the question and say "yeah close is apparently correct" - then it won't really matter if one of the votes was cast by a system – gnat Aug 2 '16 at 14:01
  • 2
    I've noticed that people on these sites tend to want to know the ins and outs of everything and how it works, so yeh community may be tricky.. but if it works, a meta canonical will soon fix the questions about it. Community is not such a bad option in some ways, as it has that authoritative feel to it. I really am not sure what the best solution is, I sometimes see questions in the queue that I don't think should be there, perhaps an option would be to bump posts out of the queue quickly, with a leave open vote? – Nobody Aug 2 '16 at 14:12
  • @Yvette you mean, make Community cast binding Leave Open for some questions in the queue? That's an interesting idea, I need some time to chew it. It would probably be harder to implement than binding close but maybe not much harder – gnat Aug 2 '16 at 14:22
  • 1
    well interestingly I actually think that's a lot of the reason the queue is so hard to manage and why I left the active organisation of killing that queue, as too many times I saw posts, by well intended users being voted into the queue. Now what proportion of the queue this is I wouldn't want to guess and that's just my observation, so I don't know what is happening elsewhere, but it makes reducing the queue to zero impossible, as these posts need to close vote age out of the queue. ... – Nobody Aug 2 '16 at 14:48
  • ... I think some binding votes by community or "trusted users" or both, to both close and "open" or bump posts out of the queue will solve the problem. and yes same type of algo but for to bump out of the queue could work. – Nobody Aug 2 '16 at 14:48
  • @Yvette I think I figured what would kill your idea of Community casting binding Leave Open in the queue. It would have only two supporting votes from human reviewers (because three votes from them would already throw question off the queue), that seems way too little. Note how I mention "3-4" human votes to close prior to letting Community cast its own, that's just more reliable – gnat Aug 2 '16 at 15:59
  • Yes I was also wondering about that. Perhaps we need to change the close vote queue to show stats by number of votes per post. So total close votes or excluding posts with only on close vote. It may help people gain perspective (or maybe it's just me that wants that). At least allowing people to easily sort by number of close votes. Though there is a script for that that some are using. – Nobody Aug 2 '16 at 16:01
3

One issue is that I lose all my close votes at a time of day that makes no sense to me, therefore.

  • Let me take forward half of my unused closed vote from one day to the next day, but never taking forward more than N close votes.
  • OR, have a “close vote account” that gets votes added to it each hour (currentNumberPerDay/24), but the maximum balance on the account is no more than the current number of close votes allowed per day.
1

This is just an idea; there are other very good answers on this page.

I know there is the reviewer badge (250 reviews, awarded once per review queue). Perhaps once that badge is reached, you are awarded more review queue close votes based on the average number of close votes that come in per day on your site. For example, on the physics.SE (which is the only site where I can work through review queues) we'll say, I don't know, 30-40 questions come into the queue with close votes. Currently, we only get 20 close votes. Once the reviewer badge is reached, you get 25% the overall average more (these are just place-holder numbers; I don't know what the exact ones would be) ending up with around 30 close votes for the review queue.

Then, the next big increase could be when you reach the gold reviewing badge (steward, 1000 reviews, awarded once per review queue). Once you reach that, you get a similar increase, also based on the average number of close votes that come in per day.

This could also be used along with other mile-markers, and maybe one or two reputation based increases (like at 15k and 30k, or something like that, as well as what the other answers suggest). The main idea of this answer, I guess, is to use the badges to increase # of close votes.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .