Should trusted close/reopen voters and reviewers have their votes weighted more than others'?

Question

In Q3 2021 I finalized my research around three vote close/reopen and I was pretty happy with the outcomes - as were the sites that got to test it out. While the benefit to each site was different, the most important thing was that there were no negative impacts of that test and all of the sites wanted to keep the reduced numbers of votes to close. We will be continuing to make this change on other sites that have requested it in the near future, so if you're waiting for a 3-vote close/reopen test, it'll be coming soon.

One of the things that came up during that process was a question about expanding the gold badge close/reopen privileges to other users as an addition or alternative to three vote close/reopen. We've discussed it internally and want to do some research before deciding whether to ask the Public Platform team to build a new feature to allow this - unlike 3-vote close, there's nothing already built that will allow a change of this sort.

This question is the second part of that research - I asked a version of this question to the Moderators in November 2021 and am following up here to see what others think, too. Based on their initial feedback, they were concerned about a few things and it didn't seem there was a huge amount of interest in this path forward, at least as proposed at that time. That said, I've adjusted the description below to answer questions the moderators had and concerns they expressed as their arguments seemed reasonable.

What is the goal?

First things first - this project is purely information gathering as mentioned in the Q4 roadmap. We're trying to see whether it's something that would be worth building at some point in the future and we have no specific timeline related to actually building it. We'll be gathering data and community sentiment before deciding whether to move forward. Additionally, this wouldn't be shipped network-wide if we did build it. Like 3-vote close/reopen, it'd be up to the site to opt-in. That out of the way, let's look at the goal of the feature...

As my research showed, there's a need across the network to make curation tasks easier to complete and require fewer people to participate. Three vote close/reopen is a way to do that and it's also a very even-handed one - essentially, everyone with the privilege to vote to close/reopen gets the same decision weight when it comes to participating in reviewing and voting to close and reopen.

In general, this is fine, but it means that we're not necessarily recognizing the specialized knowledge and experience of people who are very involved in the curation process, and I would say that recognition is deserved!

So, in the end, the goal is: Make it easier for engaged curators to close and reopen questions more quickly, for any reason. Additionally, we hope that people who are working to curate the site will feel like their efforts are more valued and that there are fewer close/reopen review tasks needing their attention.

While we can't really hope to make reviewing fun, we can hope to make it feel less futile.

Why are we investigating weighting close votes as a solution?

Up to this point, weighting votes has been limited to the gold badge "duplicate hammer" - which, as its name implies, only has any power over duplicate closures.

There are other things about the gold badge hammer that may not be a good fit for this usage:

• It's only particularly useful on sites that are huge (like SO) or have some very deep interest and expertise in specific tags (like SFF) since getting a gold tag badge is a very high bar to set.
• Having a gold tag badge doesn't necessarily mean you know when to close/reopen posts - that said, having this only relate to duplicates minimizes that risk.
• It only relates to duplicates.
• It's unilateral. This is a pro and a con - for duplicates it's likely good, but for other close reasons, it may not be as sensible.

That said, when we initially rolled out this feature, half of the answer announcing it says that we'd consider it for other uses in the future - and that was in 2014! Seems like it's about time.

Beyond this, we think this is a good potential solution for the goal we're trying to achieve. While it's not something we can start implementing tomorrow, since we'd have to get it built into the site, we do have some data and personal experience that we can look at from duplicate closures.

What are the initial feature ideas for this?

I've got some ideas floating around in my head of what I think might make a good feature set to ask for here and that's based on my personal experience and my thoughts about the duplicate hammer. I'm open to being convinced that I'm wrong about any of these points! I'll put them here in a numbered list with my thoughts so that y'all can call them out by number in your answers:

1. Increase the close/reopen vote weight to 2 for users who meet certain criteria - I think that making these unilateral has the risk of making it a bit too easy to close or reopen questions without a lot of oversight.
   • I could also go the route of making it a weight of 2 on sites with 3-vote close/reopen and 3 on sites with 5-vote close/reopen.
2. Unlocked by voting to close/reopen (including reviews) n times + - Since reopening is less common, this would get imbalanced if we relied on each action individually. We don't necessarily want to rely on reviewer badges only as they're not very common. Review stats:
   • Steward badges (1k reviews) - only 88 sites have at least one person with the badge and only 42 have ≥10 †.
   • Reviewer badges (250 reviews) - 132 sites with at least 1 and 81 with ≥10 †.
3. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.
   • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".
4. Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.
5. Applies to all close reasons - There's really no reason to restrict this to any specific close reason over the other. Except for duplicates, we don't necessarily want to incentivize people opting for one reason over the others.
   • This would not replace the gold badge hammer - so, no loss of privilege for those users when it comes to duplicate closures.

Those are the broad-stroke points. There are clearly minutiae, such as determining which close reason "wins" to poke at, but those things seem like secondary bits that we can work out once we answer the big question - "Is this idea worth pursuing?"

^{† - These badge counts are queue agnostic, so it's likely an overcount if someone has a Steward or Reviewer badge in LQP instead of Close or Reopen.}

What do I need from y'all?

Poke at my ideas, question them. Point out things that I've missed or where I'm looking at things wrong.

• Is this concept (even ignoring the specific rules I've outlined above) good/bad/silly?
• Would this be something you think the sites you use would gain value from?
• Is there something different that you think will address the goals we've mentioned?

Don't feel like you have to answer any or all of these questions. They're here as ideas for what sort of feedback I'm looking for.

Answer 1

Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.

Please don't do this. Some people close many, many more questions outside of review than in review. That should not deem their votes less valuable. Additionally, review queues on some sites are almost always empty (like here on MSE or on child-meta sites).

As an example, here are my current stats:

• On Stack Overflow, I've submitted 3,110 close votes vs 102 reviews¹ in the "Close votes" review queue.
• On Meta Stack Overflow, I've submitted 140 close votes, only 8 of which are from review.

---

¹ _{Only 80 of them are close votes.}

Answer 2

Whatever the rules are, please make them as simple as possible

People are already confused by how sometimes moderators and gold-badge holders can unilaterally close/reopen questions. Fortunately, those are both quick to explain. But here, some of the ideas you're talking about, especially if we have them interact, throw in edge case logic, etc. etc. - all of a sudden it becomes a headache to understand. How are we going to be communicating all of this?

Sure, the idea of "trusted voters" might feel intuitive, but how it's implemented may very well not be. What a "good question" or a "good question record" is, is another example of a simple concept where people are constantly being confused by implementation. Whatever the rules are chosen to be, they will need to be clearly and simply presented, or we're going to have an explosion of poor confused users to deal with.

Please remember that simplicity is a virtue.

Answer 3

I like this idea primarily because it speaks to a belief that I think you and I both share: Tying user-level moderation abilities, and the weight they carry, to non-reputation validity more easily engages users who have a track record of making good user-level moderation decisions instead of engaging users who simply post upvotable content, and overall has the potential to improve their site-cleaning efforts (i.e. the net result is a cleaner site).

Making it easier for users who have demonstrated that they know what they're doing with moderation privileges to give back to the site in a more meaningful way is something I'd like to see more of. This methodology doesn't have to replace reputation (though that could be interesting...) but it makes users who spend a lot of their time flagging, voting to close/reopen/delete, and reviewing feel rewarded by further enabling them to keep fighting the good fight in keeping the site clean.

---

So... I like the idea of weighting close/reopen votes more heavily for users who demonstrate that they're performing these actions correctly... But how we determine who qualifies is going to matter a lot, and is likely going to be tricky. Your proposal...

2. Unlocked by voting to close/reopen (including reviews) n times + - Since reopening is less common, this would get imbalanced if we relied on each action individually. We don't necessarily want to rely on reviewer badges only as they're not very common.

Having a minimum contribution for this is kinda necessary, so this is great. A concern you recognize is that not all sites have the same activity levels for a network-wide requirement for something like this, so some tweaking is probably required, but... Once you've helped vote to close/reopen enough, you should have a good idea of how it all works on the site you're helping to curate. If you haven't yet, your second criterion handles this:

3. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.

   • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

This has a lot of potential to be good. While Sonic's answer raises concerns about cases where the validity of a closure isn't always correct, I think that the grand majority of question closures will reach a consensus that overall is correct. In other words, I feel like the cases that Sonic mentions would fall into a very deep minority, and Sonic, who is quite the prolific contributor here, would certainly meet the requirements to have his reopen vote weight increased under this system so that he can help correct mistakes like the ones he mentions more effectively.

What matters the most here, in my opinion, is the threshold that we choose for "valid" votes cast, in combination with ensuring that later edits to the post that result in it getting reopened do not mark the previous close votes as invalid. I don't want to get ahead of myself here and start suggesting good % thresholds for this, and I suggest that if we go this route we open up a discussion here on Meta to talk about what a good threshold should be. I expect there'll be a lot of squabbling but I imagine the end result should be something usable. Like you mentioned, some work is required to work out a good definition of "good."

Further, in case this isn't already being considered, how close a user is to this threshold should definitely be exposed to the user. It may not need to be exposed to other users (maybe moderators?) but how close you are to this vote weight increase should absolutely be visible in some way. I don't want to have to guess at how close I am to this vote weight increase - I'd like to be able to see it as something to strive for.

Overall, I think that this idea is heading in a generally good direction, and I do feel like this has merit enough to be built with refinement.

Running through your final feedback questions:

Is this concept (even ignoring the specific rules I've outlined above) good/bad/silly?

I feel like it's certainly worth exploring. I'm a bit demotivated to hear that this proposal's initial version which you proposed to the network's moderators resulted in little interest to pursue this further, because I do feel like, if implemented properly, this could be very beneficial in keeping the site clean and acts as a stepping stone towards giving the people who show that they're good at user-level moderation expanded tooling to help out more.

Would this be something you think the sites you use would gain value from?

I primarily frequent this site (Meta SE), Gaming.SE, and a few Beta sites. While Meta SE's barrage of blatantly off-topic questions never stops and requires frequent attention (that the MSE moderators, and yourself, are always helping with, thanks by the way!), Gaming.SE certainly has some very active curators who I feel would certainly love some additional close/reopen vote weight. The site garners quite a few Minecraft questions that aren't always up to snuff and might need closed so that they can be refined, or just deleted. These questions are typically honeypots for low-quality answers that also end up being deleted after the question is closed, leading to overall confused and demotivated users. Faster closure in those cases would be beneficial.

I'm very interested to see how (or even if) this idea moves forward.

Answer 4

3. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.
   • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

Something to consider is by using a percentage of "valid" votes there is a potential snowballing of the privilege. The fact that weighting votes has the potential to make someone with the higher weighted closure vote privileges more able to maintain the privilege.

Assuming the simplest possible "valid" close/reopen definitions: "the post being closed = valid closure" and "the post being reopened = valid reopen". Then someone with the ability to more easily close/reopen a question is likely to automatically gain a higher percentage of "valid" actions.

Within review queue it is fairly easy to see if there were mixed thoughts on a particular post i.e. some users voted to leave open and some voted to close, however, outside of review queue there isn't a way to dispute an "invalid" close vote until the question has been closed.

So my follow up questions would be:

1. Should review tasks that have mixed leave open/close decisions be handled differently than those tasks with unanimous decisions?
2. Is it an issue that someone with the privilege has an easier time maintaining it than someone trying to obtain it for the first time?
   • If it is how could this be mitigated?

Answer 5

the goal is: Make it easier for engaged curators to close and reopen questions more quickly... Is there something different that you think will address the goals we've mentioned?

I would suggest to address this from a bit different angle, namely - make it easier for users engaged in other ways help curators in closing and reopening.

The only attempt at step in this direction was a red dot indicator. I think it turned into a failure, and most likely reason is that meaning of indicator is too vague to be worthy, "somewhere there are some reviews that may somehow benefit if you take some action on them - do hundred clicks to find out". Even engaged curators may find it useless and distracting - and as for other users, they probably just learn to ignore it after giving it a try once or twice.

What I would like you to try instead is something that would require people just a single click from where they browse now right into a question that has only one vote left for closure (with non-positive score) or reopening (with non-negative score). This could be achieved by giving such questions some dedicated visuals / indication in question list views.

Questions in the need for that single close vote can be indicated as grayed - just like currently system indicates answers with solid negative score. Similarly, questions in the need for reopen vote can get bolder / darker visuals. This way, even non-curators, just active users browsing the site / tags their usual way, will have an opportunity to contribute to closes and reopens with minimal effort, "just in time".

(In case if you wonder why above I proposed to complicate indication criteria by extending it with score - this is because I would prefer to make it possible for people to impact it with regular votes up and down.)

---

Above was a network wide suggestion. Now, here are some site specific notes. I am going to focus only on sites I am active at. I'll start from smaller sites. I actively participate in three smaller sites and none of them seems to have substantial need in higher weight votes. This is possibly related to scale of these sites allowing to actively involve diamond mods in closing and reopening.

I am also active at Stack Overflow and in my experience, it suffers from a bunch of problems that are unlikely to be resolved by increasing vote weight. Specifically, your focus on trusted / engaged curators makes me feel that you ignore folks who are "merely" interested in maintaining quality of content only in their tag(s).

I am uncomfortable with that because I myself started curation maintaining content only in my tags. It went quite naturally: it was manageable because I focused on just a few tags and it was easy because the tags were familiar. Also back then it seemed to be in my own personal interest which motivated quite a bit (for the sake of completeness, later I decided that general curation is more in my interest but that's maybe a different story).

Nowadays, folks willing to maintain their tags seem to be in much worse position compared to how it was when I was doing this.

To start with, close queue became inefficient because it contains either "gray area" questions which are of high enough quality to avoid triage or fairly old poor questions that were released after triage (which additionally may be answered because of keeping open for a long time, which kind of trashes the very purpose of closing). And really poor new questions hang open for too long because of being held hostages in triage. It is quite likely that this results in folks dropping out of close queue after giving it a try once or twice.

This probably discourages people from using close queue (both engaged curators and folks who would want to maintain only their tags). I suggest to fix this issue by pushing poor quality questions into close queue in more timely manner (related feature request).

As for triage itself, it is unfortunately unusable for those willing to focus on curating specific tags. I've read somewhere that triage algorithm catches about 20% of total questions. This is great, and percent of abundantly poor questions seems about right, but this also means that if someone wanted to triage questions only in specific tag(s) they would have to make hundreds skips over questions in other tags, this is just unrealistic.

I suggest to give active tag users an opportunity to participate in triage by allowing to filter it by tags, just like it is currently done in close queue (related feature request).

The last but not the least issue is huge amount of close-worthy questions in popular "big" tags. These tags are flooded by hundreds or even thousands inappropriate questions a day, which makes it doubtful that fairly narrow community of trusted curators will be able to handle such flood, no matter how much power you give to them.

Issues of this scale would better be handled by measures of appropriate scale. Specifically, I think you can try to leverage the fact that "big" tags tend to have respectively large numbers of tag badge holders and give these folks more close votes to cast in their tags (related feature request).

These folks are active in their tags anyway and it is easier for them to evaluate questions quality because they are subject matter experts and some (hopefully many) of them may be interested in stopping flood of homework dumps that pollute big tags and obscure answer-worthy questions.

Answer 6

Overall, I think this would be a good idea on those sites that have bloated Close Votes and/or Reopen Votes review queues, like Stack Overflow (which is the only site on which I have the CV privilege, as it happens).

But a few notes on some of your particular points …

There are other things about the gold badge hammer that may not be a good fit for this usage:

I agree. Linking this extra weight to tag-specific expertise doesn't make much sense, IMHO. While recognizing duplicates and providing the correct target(s) certainly requires specialized knowledge, closing for a number of other reasons does not (and cannot). For example, what of questions on SO that are off-topic because they are about general computing or network administration (amongst other things)? Who would have a gold tag-badge in a subject that is inherently off-topic? Also, some questions should be closed (IMHO) because they don't have any appropriate tags (such as a programming language).

Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.

Not so sure about this one. I know of a number of active users on SO who are very good curators but who aren't especially active in the formal review queues. Personally, I value their experience every bit as much as those who do 100+ reviews per day.

Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.

This is important – robo-reviewers should not have this extra privilege! However, the exact mechanism for defining good (as you have already said) may be tricky. There are occasions when the Close Vote queue seems to attract "pile on more of the same" type votes that are actually wrong (either in the specific close reason or the actual decision to close). You have said, We'd put some work into finding a reasonable way to define "good". – that's likely to be quite a lot of work (but important).

And a couple of short questions:

1. Would moderators be able to remove this privilege manually, if a user is seen to abuse it?
2. Would the double-weight close vote still count as only one vote, in terms of our daily allowance. (I don't see any indication to the contrary in your post, but I just want to be sure.)

Answer 7

I'm not particularly a fan of making reviews a requirement of the feature, but I could see poor review history (or even a large percentage of actions being reversed) as a way of kicking someone out of the privilege.

Another thing to consider is the kinds of users at play here. There's the heavy moderation users, and the heavy answerer users. Their habits on site will vary, as will how often they reopen and/or close. If the requirements for obtaining this privilege past having a gold badge too heavily favor one or the other, this will shift the balance in a way that going from 5 votes to 3 votes didn't.

Answer 8

4. Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.

That's the very last thing you should do. Instead of giving more weight to random reviewers who might be completely unfamiliar with the topic, but happen to have some badge or another in an run-of-the-mill tag like "string" or "image", you should give more weight to the users who come across a post naturally. I think these users will not only be more familiar with the topic, but they will also care more about the topic.

In addition to increase the weight for those users, you could, for example, also make it possible to access the now review-queue-only actions like "vote to keep open" etc. directly from the question or add direct way to go from the post itself to the ongoing review.

Answer 9

Weighted votes are problematic in some aspects and will not solve core issues.

I am talking from a Stack Overflow perspective, since I have very little experience on other sites.

My first criticism would be putting close votes and reopen votes into the same category.

On Stack Overflow there are a huge number of questions in the Close Vote Queue, while the Reopen Vote Queue is small and manageable.

Reopen Votes

Currently on Stack Overflow the number of votes needed to either close or reopen is 3. For the reopen queue this is a rather low number, because high reputation users can vote to reopen their own questions. That means only two additional votes are needed for reopening which is a rather low bar.

Also I have seen plenty of blatantly off-topic (totally missed the site) questions having two reopen votes and plenty of those even get reopened. It seems like fresh reviewers with lower reputation prefer to reopen questions (regardless of whether the question is on-topic or not) rather than keep them closed, which is another problem.

We also have occasional close/reopen wars where even gold badge holders are engaged and are abusing their power in order to gain more reputation, so they reopen obvious duplicates. Again reducing the number of votes needed to reopen questions that need to be closed would only increase potential abuse, because even more questions that don't deserve reopening would be reopened.

If some question really deserves to be reopened and there is a problem achieving that, we have the SOCVR chat room where people can ask for a question to be reopened if they think it is in good shape.

We don't have a problem with reopening questions; we have problem with closing. Any fiddling with reopening votes in sense of making it easier to reopen questions would be bad for the Stack Overflow.

Close Votes

Three votes to close question have significantly increased our ability to successfully close questions. I think three close votes are optimal for closing non duplicates, as it is not too high, but it still requires some consensus from multiple users.

Why weighted votes are not a good option has multiple aspects.

First, the main problem on Stack Overflow is not the number of votes needed to close question, but the sheer number of questions that need closing and not enough people willing to moderate the site. Weighted votes will only make a small dent and will not bring us closer to solving the influx of poor questions.

That problem can only be solved from multiple sides, including better education of new users about rules and guidance on on-topic and off-topic questions.

Weighed votes would increase frustration for new users and increase tension, because even now users complain about their questions being unilaterally closed when it comes to duplicates where the OP is pointed to another question where they can find the answer.

Closing question by single or even just two users, for seemingly on-topic, but otherwise poor or unsuitable questions, would significantly increase the friction and accusations toward close voters and we would still not be able to close all that need to be closed.

Besides questions that are just poor questions, but might be edited to shape, there is still a huge amount of blatantly off-topic questions asked and questions asked in other languages (not English).

Being able to close those questions with single vote would be extremely useful. But, weighted votes are not perfect here as they either require a gold badge holder which makes no sense for blatantly off-topic questions as you don't need to be an expert, or two people which is still not fast enough given that there is just too many of such questions.

Also, some of such off-topic questions can be migrated or closed as belonging to another site, but only for a limited number of target sites, but others fall into unclear (non English) questions or don't belong anywhere on SE.

To allow faster closing of such questions it would be far better to have a completely separate blatantly off topic or unsuitable reason that could be paired with another review queue, so that such questions still go through some additional vetting (preventing power abuse) while we would have ability to cast a single vote to close them as fast as possible.

---

If you do opt for weighting votes, please don't limit those to review queues as plenty of people close questions outside the queue. For instance, I have cast 36,785 close votes on SO and of those only 6007 from the review queue.

Additional help for solving closing problem would be prolonging vote aging periods and giving more close votes (there were days when I would easily bump into 40 votes limit, and could cast more if I had) to people that do moderate the site.

Answer 10

Here is me poking at the idea.

I see where it is coming from. It would make reviewing even more efficient, i.e. result in more reviews done with the same investment of human resources, albeit with maybe a higher error rate. Even domain experts make mistakes from time to time (and sometimes even are biased). The idea of the 3 close votes requirements and the 5 close votes requirement before that was that we wanted consensus to reduce the overall error rate. With an increased vote weight as proposed here we would have much less of a consensus effect than before.

And indeed we not only need assurances about the decision to close or not to close but also about a suitable close reason. The close reason is the most important information for the content creator, so it better be right. How do we know that we present the right close reason(s)?

I agree with the general idea (to weigh review votes somehow by trust/experience), but I see the same problem I saw when we reduced from 5 to 3 votes and will repeat it here:

We must control the error rate of reviews

Why having a vote weight of 2 for experienced users, why not 1.64 or 2.47 and a threshold of Pi or something else completely arbitrary? Obviously there should be arguments why 2 is the best and nothing else is better. The argument should be that at this weight the error rate is still reasonably low while the reviewing efficiency is already quite high.

Ideally you would measure the error rate (by letting multiple people do the same review independent from each other) and then compute a review efficiency vs. error rate relationship in dependence of experience and select an experience dependent weight that controls the error rate uniformly over all experience levels from there.

Measuring the error rate (if we assume that reviewers get it right on average and aren't biased on average) might be possible (also adding a few more false positives in the data set). This idea here only makes sense if the error rate indeed goes down with increasing experience. However, if this is the case, we might not need experience as proxy for the error rate, simply estimate an error rate for every single reviewer and take this as inverse weight. A new user could be a better reviewer than any gold badge holder. Why not? Just measure directly what you want to know.

I could imagine a system where everyone starts with weight 1 (or even zero for really new users) and depending on his review actions can either increase or decrease his weight.

And finally one more thing that needs attention:

Reduction of close-worthy content could reduce the need to review that much

Sure, there will always be content that needs to be closed and everybody makes mistakes, but a lot of duplicates might be avoided if only people searched more and a lot of unclear, unfocused questions or questions without debugging details (on SO) might not need to be closed if only the content creator would have been more careful and more knowledgeable when creating the content.

The amount of reviews and closures necessary hints that there are quite large problems with content creation that happen much before reviewing takes place.

While better education, guidance and motivation of content creators could be seen as complementary to this idea here, it affects the urgency of it and therefore should be kept in mind and resources should be spent on it equally.

This is strongly related to:

While we can't really hope to make reviewing fun, we can hope to make it feel less futile.

Reviewing can be fun (it's nothing else than giving feedback on existing content in order to improve it), if only the reviewers get the impression that the content creators invest the same amount of time and concern in the creation and improvement of existing content than the reviewers themselves invest into reviewing. If only that would be the case.

Answer 11

We don't really want to incentivize people to use duplicates when other reasons apply. I would love that all close reasons behave like duplicates close reasons. If I can be trusted to know that two questions are the same, I also can be trusted to know that a question is unclear, is not within the site defined topics, is too broad or opinion based.

I don't really want the system to be more complex than it already is, I want to know that if I cast my vote I know exactly what will happen: either the question is closed or my vote goes towards the pool of votes. Being surprised that my close votes behave differently in different context makes zero sense to me. If I cast a vote, wherever I cast it I want it to behave the same way.

Answer 12

there's a need across the network to make curation tasks easier to complete and require fewer people to participate.

Giving some individuals more power over which questions get closed or reopened is one way to do that. Another way to get more done with less manual labor is to automate some of the work.

There is going to have to be some effort to determine how someone's vote will be weighted more than someone else's and by how much, so maybe some of that work should be to determine if there are ways to off-load some work to an automated system. For example, (and this is just an example to explain what I mean, not something that I want to debate the details of) if a post has certain characteristics that are indicators of low quality posts, maybe one close vote by anyone is enough to close it, but it automatically goes into the reopen queue for confirmation. Quickly closing a low quality question prevents well-intentioned bear feeding and the reopen process tends to be more amenable to salvaging questions than the close process.

I think it is a mistake to weight close votes solely by who is casting the vote and their close-voting expertise, and not by the nature of the content they're voting on. The reason some sites still need 5 close voters to close a question isn't because the people reviewing are somehow untrustworthy or incompetent, it's because the content is harder to definitively and objectively separate into off-topic and on-topic and the queues aren't getting so overwhelmed that we need to reduce the quality of the reviews to keep up.

Answer 13

Thanks for posting this! I have quite a lot of thoughts... no guarantee they're all included here in the first pass:

There are other things about the gold badge hammer that may not be a good fit for this usage:

It would be good to revisit the topic of expanding gold badge powers as its own topic, but since you brought it up as a comparative example applied to this proposed feature, here are my thoughts:

• It's only particularly useful on sites that are huge (like SO) or have some very deep interest and expertise in specific tags (like SFF) since getting a gold tag badge is a very high bar to set.

You mentioned above this wouldn't apply to all sites, only those who want it. How many would want it? We need some data on that, so we can see if there are actually any sites that would want it but would have trouble getting it. If so, that can be part of the "improve the gold badge holder close feature set" discussion... doesn't seem particularly relevant here.

• Having a gold tag badge doesn't necessarily mean you know when to close/reopen posts - that said, having this only relate to duplicates minimizes that risk.

While true, there are other, non-mutually exclusive alternatives between the two extremes of "everything" and "only duplicates". Wouldn't a user who is considered a "subject matter expert" in a tag know if a question using that tag is, say, an opinion-based one that can't be objectively answered? The "opinion-based" reason seems like quite safe low-hanging fruit in terms of expanding the privilege. Why not start there?

• It only relates to duplicates.

This is only so because Stack Overflow made it so. You could easily apply it to all close votes, I suspect. Certainly if you're applying the powers to another set of features, some changes would be not only opportune but also appropriate!

• It's unilateral. This is a pro and a con - for duplicates it's likely good, but for other close reasons, it may not be as sensible.

Again, this can easily be changed.

---

3. Requires maintaining n% of close/reopen votes & reviews deemed "valid"

While I agree that preventing robo-reviewers from having additional, more powerful abilities is ideal, I think we need more data (on users' vote reasons and how they tend to spread their votes across reasons, especially in the review queues) before we can discuss this to a significantly deeper level. In the review queues, I tend to spend most of my time narrowed by close reason or by topic. Some close reasons apply to questions that are permanently or inherently off-topic... a request for recommendations on which language to use for a project is never going to be on-topic. While it's not likely to happen, it does sometimes occur to OPs of such questions to delete the entire question and rewrite it as a new, totally unrelated one... which might well be on-topic.

Likewise, and far more important here, are the close reasons that are inherently temporal in their application: the "needs detail", "lacks MRE", and "needs focus" reasons (or as I like to call them, "Unclear", "No MCVE", and "Too Broad" reasons... seriously, could we get those old naming/descriptions back?). These close reasons can easily be invalidated (which is great!) by an OP who diligently edits their question to narrow the scope of it, add missing detail, or include a necessary error message/stack trace.

Such "how it is now"-dependent close votes are correctly cast in the moment, and should not be discouraged. Yet, if we start to pin future, powerful abilities on surpassing a certain threshold of "successful" votes... you might punish users who've cast these votes, or worse, discourage users from casting votes of a certain type for fear of shooting themselves in the foot.

I see two ways, of the top of my head, to help combat that problem:

• Only count vote success for questions where the question has not been edited (by someone other than the voter) since the vote was cast.

And/Or

• Only count vote success for questions that have been successfully/completely voted upon (e.g. for close votes, the question was either closed at some point or left open via review completion or all close votes aging away; same appropriate adjustments for reopen votes) and some time-based constraint such as:

  • where the vote was cast in the last 365 days.
  • the above, but vice versa (before the last 365 days, instead of since 365 days ago).
  • where the last status change of the post (closed, reopened, edited by user other than the voter) was more than 365 days ago.
  • the above, but vice versa again

(I provide conflicting options here because I've not thought enough yet about which one would be better)

---

Unlocked by voting to close/reopen (including reviews) n times +

I think this is a good metric, but it doesn't have to be static. I would tend to think more reviews are better, but 'age/seniority' is a stale metric by itself, in my opinion. One alternative would be to count "reviewing/voting to close or reopen n times or more in the last 365 days.

This would be unique in that it would be a time-based requirement that could be lost and regained, based on your reviewing activity. (E.g. "Haven't performed close/reopen moderation in years? Well, you'll need to get 200 (for example) close/reopen votes under your belt this year before you have it again" ... which is only 4-5 full days of voting or reviewing). This could be implemented via a sliding time requirement or factor in whether someone had it at the end of a year to grandfather them in at the beginning of the next year, to avoid people who have it suddenly losing it come New Year's Day.

Of course, it could be combined with a flat requirement of n close/reopen votes altogether, to ensure a minimum level of experience/exposure to the process and privilege overall.

This may be too complicated, or lead to too much confusion ("hey, I had double-weighted close votes in the past, but don't have them anymore!"), but I think it's worth considering. It would certainly help ensure accuracy by ensuring recent familiarity with the process... since content standards change over time on a given site (what was OK to ask in 2009 is often not OK to ask in 2022).

---

We don't necessarily want to rely on reviewer badges only as they're not very common.

On that subject, this is again problem a topic best discussed separately, but why not visit the idea of weighting badge requirements based on site activity, like you already do for Beta vs non-Beta sites? Either total questions, or questions per day... start out with lower thresholds for recently-graduated sites that are above Beta but below what Stack Overflow (for example) has, and once a site hits the threshold(s) the first time, it changes to the full/highest threshold and stays there permanently (to avoid it constantly changing back and forth for sites who hit a questions/day metric one day and then don't the next, etc.).

Only weight votes cast in review

As others have said, I think this is bad; many, many user moderation actions (most, even?) are done outside the review queues. Rewarding only users who perform them from a /review/ URL is unfair and would miss out on a huge chunk of users actually using the feature.

Answer 14

3. Requires maintaining n% of close/reopen (...)

   "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

If we're talking about the scale of Stack Overflow the overall number of successful close votes should outweigh the usually small percentage of reopened posts. A lot of closures fall into routine maintenance so lets say a user that cast x number of successful close votes (somewhere in the several thousands) can be trusted to close their daily quota at increased weight because a bulk of the daily posts are straightforward to evaluate.

In other words, focusing on percentage only overlooks the fact that a close voter with 10k CVs at 95% remaining closed is more experienced and effective than a close voter with 1k CVs at 99%. (The 10k reviewer is 10X more effective at only a marginal 400 post difference to a 9000 reviewed post gain.)

EDIT:

I'm going to include two comments because I think they are relevant and I want to raise a use case as counterargument:

by Catija

I'd likely be considering recent participation, so either votes cast in the last n days or the most recent n votes rather than looking at someone's all-time stats.

and

by Akixkisu

They will also have no issue quickly regaining that trusted user status — since they can do their daily quota every day

The use case that comes to mind as a counter-example is the tag expert who's only engaged in one or two tags and tends to them daily (maybe without having achieved a gold badge). I see a lot of those CVs aging away and with the short sliding window those SMEs are likely to not see any change to their CV weight.

Answer 15

One option is to give "2 vote power" to both close and "don't close" to Silver/Gold tag badge holders so people can look after the tabs they monitor new questions on. I think this should only be done outside of review queues, as people tend to read questions more carefully when deciding if they will answer the question.

(This does include enabling a silver/gold badge holder to vote to not close before the question has been closed.)

Maybe a "leave open" review response should convert such a close vote into a single powered close vote, so the additional power only effects the most clear-cut cases.

Answer 16

Catija, you may remember me (or perhaps not) as having recently requested three-vote-close on Ask Ubuntu. I've since changed my mind on this, after having hit 3k rep myself there and seeing the (IMHO) poor quality of close-vote reviews there.

Unlocked by voting to close/reopen (including reviews)

Two problems with this:

• First, the number of reviews someone has done does not make them an expert in all topics on a particular site. Even gold badge expertise isn't a good indicator. The knowledge needed to judge (and answer) a question is often not based on "general knowledge" in a subject area, but specialized knowledge in the topic being discussed.

  For example, a gold-badge holder in the "Windows" tag on Super User may (but more likely, may not) have expertise in (to take a random example), Gimp. A Windows Gimp question may even appear unclear to them, but perfectly clear to someone strongly familiar with that particular application.

  The same goes for someone who has voted on 1000 posts across the site. That should give their review of a Gimp question no more (or less) weight than anyone else.

  And this doesn't seem to be a "corner-case" to me. I very often find "Needs clarity" close-votes on posts that are perfectly clear to me (or at least salvageable by edit).

• The second problem is that this just makes people rush through reviews even faster, further reducing the quality. For instance, voting to close because a "quick read" of something is "needs details or clarity" is far easier than editing it to make it more clear and readable. I rarely hit my review cap during any given day primarily because I spend more time editing (and sometimes even answering) posts there that others have been VTC'ing as "Needs details or clarity."

  Encouraging reviewers to "rush" their reviews in order to reach "trusted reviewer" status more quickly clearly has the side-effect of reducing the time spent and (by extension) the quality of those reviews.

Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.

I agree; I just don't think there's any reasonable way to do this algorithmically. No matter what you attempt to measure, it's likely to be gamed.

"Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

For instance, "voting with the masses" seems to be a solid bet -- If you VTC on something that already has 3 or 4 votes, it's very unlikely to be "overturned" unless it's a contentious post (close/reopen/close cycle), which is pretty rare on most sites.

Again, in this case, it's far easier (and faster) to vote than to attempt to answer or edit, even when warranted.

Answer 17

So long as it increases the diversity of reviewers, I'm onboard

Unfortunately, by the sounds of it this might achieve the opposite. Or not. It's hard to tell without more specifics. Maybe it pulls people into review if they could do more reviewing.

We have a handful of core reviewers who while appreciated, narrow down the number of people who see each review and disincentivize new reviewers starting. People are already asking Why are the review queues always empty?.

Is there any way of looking at how many people will be granted extra privileges under each of the options? I'm struggling to follow the various options.

Answer 18

"What are the initial feature ideas for this?"

I have one addition, try to keep the logic simple both from a user and programming perspective. In or out of queue should be equal (much like the gold tag hammer), and I do agree (at least on Meta) that it should apply for all close reasons:

• One gold tag badge, or
• Two silver tag badges, or
• One silver and two bronze tag badges, or
• Three bronze tag badge holders can close a question; otherwise
• 5 users, but 4 with one of them bronze and 3 with one of them silver.

Answer 19

I think there's possible low hanging fruit. On a site with only three close votes needed to close, you could look at a gold tag badge duplicate closure as being a close vote with a weight of 3. From that angle, it kind of seems intuitive (to me- at least) to give people with a silver tag badge the privilege to cast close/reopen votes on posts with that tag with a weight of 2, and for bronze tag badge holders, with a weight of 1. And I'd propose that this not be limited to duplicate-close/reopen votes. Having the rep-based close-vote privilege would work basically the same as how it works already, and for the purposes of daily vote limits (50), casting a close vote with any weight would still only subtract 1.

I'm not suggesting that this would fully solve the problem (there definitely is a large problem on SO of posts that should be closed not getting closed fast enough or at all). I find that depending on context, getting any level of a tag badge is really difficult (some tags are just really low traffic, and you may be "competing" with others who just have way more expertise, though you have enough expertise to discern when most questions in the tag require more detail/clarity to give a good answer). So I do think that having an alternative route to higher closure privileges that involves helpful closure flags / votes is a good idea. I'm just throwing out a starting idea/step that's been on my mind for a while.

Answer 20

Please make this a per-site setting

Some sites might want this, others might not. While I do realize that this slightly contradicts making it as simple a possible, there are some sites which have questions often get bounced between closed and reopened. This would only increase the problem. Also, I don't see any possible downsides from making this a per-site option, so please do so.

Answer 21

When it comes to the specific criteria for part 1, counting users' close votes as multiple votes if they meet certain criteria: using the same criteria of gold tag badges (or tag badges in general) is likely not a good idea since the tag badges only count one's skill in answering questions, not asking questions.

For duplicates, basing the additional weight on answering skill makes sense - closing a question as a duplicate is basically the same as answering the question with a link, without making a statement as to whether the question is a good fit for the site or not. Users who've answered questions within the tag are more likely to know where the relevant answers to a question are, so the current hammer criteria are sensible.

On the other hand, for other close reasons, I think that the best metric to determine additional weight is one's skill in asking questions. Users who have experience in asking positively-received questions on the site are more likely to know what's a good fit for the site and what's not, since they have firsthand experience with having their questions curated by other community members. As an example, if someone's asked a question that fell just short of the guidelines, had their question closed, asked about it on the site meta, and got an explanation as to why their question isn't a good fit, the user is more likely to have an eye for future such questions.

I don't think it's a good idea to base the privilege on number of close/reopen reviews. If you are to do so, as pointed out in another answer posted while I was writing this, there are users who are heavily involved in casting close votes on questions directly ("external" votes) but don't really use the review queue. Such users would be shut out of the additional weight even though they've demonstrated casting close votes properly, which goes against the purpose of your idea.

Doing so based on "correctness" of votes is not likely a good idea, either. Often, I come across questions that users are incorrectly voting to close or reopen because of a mistaken assumption (e.g. a question that simply mentions SO that others are voting to close as site-specific, that is actually not). Even after I review to leave open, leave a comment pointing out it's on-topic, or even in some cases edit the question to make it clearer it's on-topic, the question sometimes still gets closed, probably because users follow on to the existing votes. In that case, my review would get counted as "incorrect", even though it was technically correct (the question was subsequently reopened/re-closed).

In summary, the additional weight for voting to close questions shouldn't be based on review or vote history, but instead on demonstrated knowledge of asking questions, just like how the current duplicate hammer is based on demonstrated knowledge of answering questions.