Is using reputation to scale the number of close votes the best way?

Question

When we were working on the last se-quality-project, I wanted to introduce something into the review system that I still feel is pretty lacking. There's no way to really track your efficacy as a reviewer.

For instance, how many of your close votes actually resulted in something being closed? How many posts that you worked on in the helper queue actually went on to do rather well? In other words, how many times did the action you prescribed actually turn out to be the best action to take?

It didn't go in because showing these kind of stats to folks is difficult; it's hard to present them in a way that isn't critical when you're trying to be encouraging. It was also a lot more work than I anticipated.

Anyway, my point - we've been talking about the idea of scaling close votes based on the amount of rep you have. I don't think that's a bad idea fundamentally, but I have some concerns:

• Really good reviewers also often turn to reviewing, moderating and janitorial stuff as their primary way of participating. It's great if they have gobs of rep when that happens, but not everyone does. I barely coasted to 20k on Stack Overflow just from 'interest' on my old answers once I became more interested in the human aspects of the site.

• It would give a lot of people more votes, and I don't know if that's what we really need. What we really need is for the folks that have honed their review and moderation skills more votes to work with.

I'd like to open a pretty broad discussion just to get input and ideas. I really feel like we should be giving people more votes based on how accurately they tend to cast them, but I'm also not opposed to giving some additional votes as people reach notable rep levels.

So let's say both ideas are open, would you award 5 extra close votes after 15 questions the user voted on were ultimately closed? More / less? Should 10k users get an automatic additional 10 votes provided that they've voted to close at least 10 questions successfully?

It's really easy to get kind of complicated here, so I'm open to broad ideas. We're going to be starting the second iteration of the quality project in the not too distant future, and I'm hoping to nail down at least the first 10 things that will be in scope (and this is one of them).

Answer 1

This answer is entirely from the angle of a primarily dedicated close voter on SO. My version of that, myself, is one in which only a few questions seem intellectually intriguing per day to want to answer. So it works out well. Mostly because I can earmark the question until later, that being a question not subject to FGITW anyway.

We have the SO Close Vote Reviewers chat room that performs community moderation in a larger group of peers reviewing questions. The SOBotics chat room for various experimental bots searching content. And the Campaigns room for one-off occassional drives to close the close-worthy for whatever tag someone desires (hoping they provide the reviewers too). Naturally, there are other rooms. Petter from SOBotics has a bot posting links to other rooms quite focused on their thing.

In Campaigns a few days ago we close to tripled the closes in one day for c#. Bots do a lot of filtering prior to our eyes and brains needing much work to spot things. Mainly by downvote and close vote counts. As such, we are able to participate across many tags each day even if they are not in the tags of our immediate expertise or comfort zone. Frankly, most of those questions are just begging to be closed.

The problem is, we run out of votes. Note also that much of this is performed outside of the close vote review queue. Some of us work off of lists and do not have the mental anguish of "is this an audit?" I can't recall the last time I was in a queue.

In an ideal world we would not be called upon to do this in such small numbers of actual participants. Close voting, afterall, doesn't pay much rep now does it. Yet, there is nothing keeping us from sanely reviewing up to 75 or more per day on the days we are so willing. Like in Campaigns, but not everyday. Close voting does tax the brain.

So what do I suggest? Allow for vastly expanded close voting privileges for those pro close voters that have this interest, skill, track record. There is no guarantee we will use it everyday or even 30% of the time. We do try to find some balance in our lives and pursue other things, afterall.

Answer 2

If the goal is consistent, high quality content? Then yes, by all means make it easier to close and ultimately delete/remove low quality stuff.

My recommendation is to analyze the huge quantities of existing data SE has to identify what the characteristics of good closer voters are. Some suggestions:

• How many users have 90% of the questions they vote to close remain closed without an edit
  • Filter out questions that go close vote -> edit -> reopen/not close to not get counted against users
  • What are characteristics identifying "good close voters" and "bad close voters"
• Are there higher thresholds than 90% you can identify?
• What are reputation profiles for people who actively participate in closing, editing, and review queues? Does higher reputation correlate?

It would be much better to use more intelligent thresholds for something like close votes than "merely" reputation. Maybe make a "close weight" score. Every successful "valid close" adds 1. Every invalid close subtracts X. I don't know exact details but something like that.

I also think that the circles for "high rep" and "active reviewer/closer" don't overlap much. So if the goal is equipping people actually doing the closing/reviewing to better do so, make sure that reputation is actually a meaningful lever.

The bright side is if someone figures out the above bullet points they will know the answer to those questions, too.

And last, an important hypothetical question, is someone with 30,000 reputation and 0 close votes better equipped to close questions than someone with 5000 reputation, 5000 close votes cast, with 3000 reviews completed?

Answer 3

Like others, I'd tend to deemphasize reputation as a factor in this. Using the percentage of close votes that result in closures that stay closed is a good idea, as enderland suggests. Here's one more idea:

Factor in the ratio of close votes vs. leave open votes on questions I vote to close.

Why is this a big deal? Consider these scenarios:

1. I cast the initial vote to close on Stack Overflow, and no one else votes before my vote expires, so the question stays open
2. I cast the initial vote to close on ELU, and three users after me vote "leave open."
3. I cast the initial vote to close on a beta site, three users after me vote to close, and no one selects "leave open," but my vote expires before the question is closed.

In (1), you don't know much about the accuracy of my close vote. Maybe it was appropriate, but maybe not. In case (2), it's pretty clear that my close vote was probably a bad one. In case (3), on the other hand, it's pretty clear that my close vote was appropriate.

Thus I'd suggest emphasizing the ratio of close votes to leave open votes. A metric based on closed questions that stay closed may be useful in conjunction, but I think the ratio I've described is likely to be more indicative of appropriate close voting.

Answer 4

I agree with many of the answers above: people who have high rep usually aren't the people who are doing moderation. For instance, on my main site, Arqade, I find myself doing lots of reviewing and editing, but not much answering. Hence, I have hovered around 4k rep for quite some time.

---

But, the main thing I want to say is: if you do base the number of close votes on previous accuracy, make sure you know the right formula to do so:

See the linked article for more details, and implementations in Ruby and SQL. (Dislamer: I have nothing to do with this blog or article.)

Note that the article is talking about positive and negative ratings; you can simply replace positive ratings with useful close votes, and negative ratings with incorrect close votes to achieve the same result.

What's wrong with a more (simple) formula?

There are two formulas which are commonly used (some of which have been already suggested in this Q), but are not the best formula for the job:

1. Score = (Useful CVs) - (Incorrect CVs) - works well if all users have cast the same number of CVs, but if someone has cast a large amount more than the others at a higher error rate, they may still be considered a 'good CVer'.

   Example: User 1 casts 600 CVs. One third are incorrect. Score: 400 - 200 = 200. User 2 casts 100 CVs, all are correct. Score: 100 - 0 = 100. According to the formula, User 1 is better, but we know that is not the case.

2. Score = Average rating = (Useful CVs) / (Total CVs) - works well if all users have cast many CVs, but gives a deceptive result if a user has not cast very many CVs.

   Example: User 1 casts 500 CVs, only 10 incorrect. Score: 490 / 500 = 0.98. User 2 casts 5 CVs, however none were incorrect. Score: 5 / 5 = 1. According to the formula, User 2 is better, but in reality, User 1 is probably better, because they have more experience.

Why this formula is better

The above mentioned formula incorporates both the proportion of good-bad CVs, and the uncertainty caused by a low total number of CVs, which means that it covers the weaknesses of both of the above mentioned examples. Yes, you could simulate this by adding a second or third requirement (e.g. you must have cast at least x close votes at y% accuracy within z amount of time), but the above formula (should) be easy to implement with little tweaking.

Answer 5

Most people don't use all their close votes, and on many sites you couldn't responsibly use 20+ close votes in a day anyway, unless site scope just changed and a bunch of cleanup is needed. And as others have noted, high rep isn't a good predictor for reviewing activity. So I think we want some sort of adaptive system, not a blanket "ok, make that 30 (or 40, or 50) instead of 20". We should instead look for ways to give effective reviewers more votes, as other answers have noted.

Definition: a disputed close vote is one where the question was not closed (reviewers disagreed or vote aged away) or where the question was closed but then reopened without edits.

My proposal is in the spirit of our other rolling limits. If, in the last $time_period, somebody used all his daily close votes at least $days times and had fewer than $disputed% disputed, increase his limit by $extra_votes. If $disputed% ever reaches $too_many_problems, reset to the default number of close votes.

The exact values should be tuned by data analysis, but maybe something like this: If, in the last 30 days, somebody used all his daily close votes at least once and had fewer than 10% disputed, increase his limit by 5. If the (overall? recent?) proportion of disputed close votes reaches 25%, reset to the starting number of close votes.

With a scheme like this, an effective close-voter gradually gains more votes the way an effective flagger gradually gains more flags.

As I said, the numbers should be tuned. But the idea is that so long as somebody is (a) voting well and (b) bumping up against the limit, gradually raise the limit. If those votes aren't being used well, reset. (Possibly also start aging away the extras if they aren't being used. I haven't thought through that scenario.)

Answer 6

To expand upon the current answers.

I do not think giving more close votes alone is the optimal way to go. I think close votes should become weighted with either rep or tag badge progress (although tag progress can be slow).

Agreeing that reviewer fatigue is an issue as pointed out in this answer with these results.

To answer I am proposing an additional option. This algorithm would require two factors:

Part A

So to use the type of algorithm you are proposing on increasing number of close votes I would recommend that when a user reaches 10K rep that the close vote counts as two close votes.

So a question would require:

• 5 close votes from (user <10K)
• 3 close votes from (user <10k) + 1 close vote from (user >10k)
• 1 close votes from (user <10k) + 2 close vote from (user >10k)
• 3 close vote from (user >10k)

This could be tiered to increase the close votes for users > 20K to count as 3 close votes.

• 2 close votes from (user <10k) + 1 close vote from (user >20k)
• 1 close votes from (user >10k) + 1 close vote from (user >20k)
• 2 close vote from (user >20k)

So close votes would carry (to coin a phrase) a weighted count or close vote count.

To close a question requires a total close vote weight count of 5.

Close vote counts:

• A question requires 5 close vote counts to close (except dupe hammering).
• A user with rep >3K and <10k has a close vote count of 1.
• A user with rep >10K and <20k has a close vote count of 2.
• A user with rep >20k has a close vote count of 3.

Part B

As not all users are equally active in moderation, there needs to be a check to ensure quality of close vote weights.

So the above alogrithm needs to be inclusive of the following algorithm.

This should remedy the close review queues more quickly than increasing the number of daily votes.

For each 1K of rep the voter must have minimum 100 close votes cast that have resulted in the question being closed, let's call it a benchmark of close vote success.

Benchmark of close vote success:

• Close vote weight of 1 does not require a minimum benchmark of close vote success.
• Close vote weight of 2 requires a minimum benchmark of close vote success = 1000.
• Close vote weight of 3 requires a minimum benchmark of close vote success = 2000.

By not including a percentage of successfully closed questions over total close votes, we are not penalising people for reviewing in less popular tags of discouraging people from reviewing at all. We don't need to do that.

It does, however encourage users to vote to close questions that already have close votes rather than voting new questions into the queue, if they are concerned about the close vote success benchmarks.

Answer 7

As elaborated in some other answers, few people use up their close votes on a regular basis and thus giving more close votes to those users who already have some is unlikely to help. I already argued in my answer on Proposal to make close votes scale with rep that the logical step would be to grant the close privilege to users who do not have it already.

While my answer to that question was based on reputation, there are other reasonable criteria to award close votes to users below 3 k reputation:

• Reviews in other queues, in particular reviews of questions in first posts (as those pertain to the task of a close reviewer). For example, for each 20 questions you reviewed in first posts, you gain a close vote. To ensure that you reviewed carefully, one could deny this close vote if:

  • you failed review audits;
  • you cast too many upvotes or no action needed on posts that were later closed (or, even worse, deleted as spam/rude/abusive).

• Successful close flags, i.e., helpful close flags. (IIRC it’s a subsequent close vote that makes such a flag successful, which makes sense as it accounts for edited questions.) To ensure a minimum familiarity with the site, it seems reasonable to put some reputation threshold on this, e.g., 500 (the reputation needed to review first posts).

As an alternative to keep things simple and avoid confusion, one could simply count each flag for closure like a close vote if the flagger meets certain criteria similar to the ones listed above. The only disadvantage is that some low-reputation users with a high number of flags (e.g., from flagging spam or comments) can cast a lot of close votes this way. However, should they abuse this privilege, they also cast unhelpful flags and there is already a system in place to block them.

Finally, as an icing on the cake, it also makes sense to apply similar mechanisms for users above 3 k. E.g., if you performed a lot of first-post reviews on questions, you gain some bonus close votes.

Answer 8

I don't actually think the number of close votes people have is a problem. The idea of blanket more close votes has been attempted, and deemed a failure (together with review queue limit increases) before.

Shog9 said in his answer here that very few people even regularly hit the cap, so I don't think just giving them more votes will make an appreciable difference.

TL;DR: I would like to see some data on wether or not higher rep users are actually reliably hitting the cap before thinking that it would be necessary to give them more.

That said, reviewing 40 close votes a day is ... draining. Not many people do it, and the ones who do don't do it every day. Unless you're organizing via chat, I can barely see many people naturally hitting the close vote cap, unless they're on SO all day.

The matter of fact is, the core problem on SE sites is that closing bad questions is a purely altruistic task, while answering them instead usually gives you a bunch of rep. Its the reason why people concerned with quality often flatline around 3-4k rep. If you have the close votes and care about quality, its quite hard to find stuff to answer at reasonable speed.

So long as the direct (reputation) incentive is actively discouraging you from closing a terrible question over answering it, closing questions will always be an uphill battle: You'll have to fight the askers who don't like their question closed and want answers and you'll have to fight people who want quick points for answering the 19500th dupe question "re: What is a Nullpointerexception in java?".

Answer 9

I hear those on the "Overall rep != Quality Reviewer" side, but the simple facts are

1. You can't just shotgun rep (except Jon Skeet). There's a cap per day so you have to earn it over time. There's a reason 20k users are uncommon. Is there a chance someone might just "coast" to 20K? Sure. But that would be something done over years. And if you're not actively contributing over that time and hit 20k anyways it's not likely you'll be interested in moderating anything.
2. We tie moderation to this anyways. The best example here is the gold badge Mjölnir. One shot close powers for duplicates. If we can't trust a bare minimum 10k user with that kind of power, whom can we trust?
3. It still takes 5 votes to close anything else. And Closures aren't permanent (hence the Reopen queue). So on the off chance people get things wrong, other people can get it right.
4. You get more flags the more flags you have accepted. You always get 50 close votes no matter if you have 3k and started yesterday or 300k and 40k close votes (now that's dedication!)

If we can't trust someone who has hung around here for years and has hundreds of upvotes, who can we trust?

The flipside?

1. SO gets ball-parkish 6k questions a day. A lot of them are in terrible shape, asking things that have been asked before or don't deserve an answer.
2. Professional Low-Quality Reputational Workers will often troll these questions in hopes of some quick reputation. So the questions get closed, but not quick enough to make a difference.
3. The answer gets an upvote and thus the Roomba is out. So now you need to get 2 other (10k+) people to come back and vote to delete it. Or you just let it hang out there.
4. Reputation is gained if you just leave it.

I did a search the other day for a common PHP problem. I found some 2400 questions that appeared to be asking the same thing with nearly 1000 that were closed. I can't help but think that some of those might be deleted now if it they had been closed quicker.

So I say more close votes for more reputation.

Answer 10

^{(Estimating reliability of the voter is addressed in separate section at the bottom. The rest of this answer assumes that there is a way to have it.)}

What we really need is for the folks that have honed their review and moderation skills more votes to work with.

If you want to leverage skills of reliable close voters and can somehow estimate these consider "teaching" the Community ♦ user to cast votes to close. I mean, make it cast a binding vote on a question that has enough close votes (3 or 4) from voters who are sufficiently reliable.

Clear benefit of this approach is that it is strictly focused on what you want and doesn't involve risky changes that could lead to hard to control side effects. You experimented with 3CV at Programmers and with 60 reviews at SO and every time you were getting undesirable side effects, and you don't really know if any other indirect way to leverage voters experience turns good enough.

Directly "converting" voters experience into Community closes has minimal side effects and you can easily tune and tweak the criteria that triggers it, for example exclude duplicates, questions with many views, positive score etc. If needed you can set specific per-site criteria or even completely turn it off.

^{One difficulty here could be how to break ties on close reasons. For a start, Community could abstain of casting a vote when there is no clearly winning reason. Maybe if you choose to account for "reliability score" of the voters when resolving ties this will become a corner case.}

^{Another thing to take care of is to ensure that Community votes do not count when you estimate reliability of the voter. This is to avoid an effect of self-fulfilling prophecy. "Community cast a vote because I am known as reliable voter and this additionally increased my reliability estimate" - do that 100, 1000, 10000 times and your estimate may seriously skew.}

---

Speaking of suggestion that triggered this discussion, to scale close votes based on the amount of rep, I would want to share my understanding of what people wanted in this request (and many similar ones). It is way too often that I see this interpreted as people willing more questions to be closed (and subsequently translated into concerns of how to handle more closures).

The way I see it, it's really different. People who complain just want to hit close limit less (or get too close to it). They don't really want more closures, they hardly care about this.

This may sound like a subtle difference but I think it is important. If my understanding is correct then the way to address the issues raised in requests like this is to figure category of users experiencing this kind of issue and find out how system can (if it can) grant them extra close votes with minimal impact and minimal risk.

^{And when I say minimal impact, I mean just that. Don't try to leverage extra votes obtained this way (at least until you gain sufficient evidence that these go the way you want) - instead try to make sure that these make minimal impact. For example, if you expect or observe significant increase in closures, you can try hedging this somehow, say by making votes expire faster. Focus solely on making users happier without breaking the system while doing this.}

First thing you need if you pick this way is to decide what kind of users experience the issue. Does one who bumps or gets too close to the limit 100 days in a year qualify? Does one who bumps once a week? once a month? This is important because the better you estimate them, the better are your chances to get rid of complaints with minimal impact of the change.

Next thing is to pick category to best cover the group of users you want to help. Are they mostly active reviewers, or tag badge holders, or high-rep users etc? Picking the right category and narrowing your target users is important, again, to minimize the impact of the change.

The last but not the least is to estimate how reliable are close voters in the category you picked. This is critical because this defines how you will handle the issue.

If they are generally reliable, you grant them extra votes. If they aren't... well, you decline the request.

^{Sorry, we would be happy to give you extra votes but our stats indicate that most users experiencing this problem tend to vote wrong. We have strong reasons to expect that giving extra votes will lead to increase of incorrectly closed questions.}

---

How to estimate close voter reliability

I believe that there would better be a dedicated, separate discussion focused solely on that. There are just too many topics to be addressed to properly fit it here.

For example,

• Is it safe to only use votes to close for simplicity or other related actions need to be taken into account - reopen, Leave Closed, Leave Open?

• Is it safe to ignore deleted questions for simplicity? (I bet it isn't but put it here explicitly to make sure it isn't lost)

• Is it safe to ignore expired / retracted close votes for simplicity?

• Is it safe to ignore edited / reopened questions for simplicity? (at SO it probably is but what about smaller sites)

• Is it safe to ignore close reasons for simplicity? Is a voter who cast 1000 "unclear" votes on 1000 questions that were closed as "too broad" reliable?

• Is it OK to ignore question score for simplicity?

• How to account for review audits? What about disputed audits - those that become ineligible due to human factor correction?

• Should "reliability score" be displayed publicly, or only to the user, or kept internal? What about "concerns that it would encourage problematic behavior"? (fate of the accept rate suggests that these are better taken into account)

Answer 11

One issue is that I lose all my close votes at a time of day that makes no sense to me, therefore.

• Let me take forward half of my unused closed vote from one day to the next day, but never taking forward more than N close votes.
• OR, have a “close vote account” that gets votes added to it each hour (currentNumberPerDay/24), but the maximum balance on the account is no more than the current number of close votes allowed per day.

Answer 12

This is just an idea; there are other very good answers on this page.

I know there is the reviewer badge (250 reviews, awarded once per review queue). Perhaps once that badge is reached, you are awarded more review queue close votes based on the average number of close votes that come in per day on your site. For example, on the physics.SE (which is the only site where I can work through review queues) we'll say, I don't know, 30-40 questions come into the queue with close votes. Currently, we only get 20 close votes. Once the reviewer badge is reached, you get 25% the overall average more (these are just place-holder numbers; I don't know what the exact ones would be) ending up with around 30 close votes for the review queue.

Then, the next big increase could be when you reach the gold reviewing badge (steward, 1000 reviews, awarded once per review queue). Once you reach that, you get a similar increase, also based on the average number of close votes that come in per day.

This could also be used along with other mile-markers, and maybe one or two reputation based increases (like at 15k and 30k, or something like that, as well as what the other answers suggest). The main idea of this answer, I guess, is to use the badges to increase # of close votes.