54

In Q3 2021 I finalized my research around three vote close/reopen and I was pretty happy with the outcomes - as were the sites that got to test it out. While the benefit to each site was different, the most important thing was that there were no negative impacts of that test and all of the sites wanted to keep the reduced numbers of votes to close. We will be continuing to make this change on other sites that have requested it in the near future, so if you're waiting for a 3-vote close/reopen test, it'll be coming soon.

One of the things that came up during that process was a question about expanding the gold badge close/reopen privileges to other users as an addition or alternative to three vote close/reopen. We've discussed it internally and want to do some research before deciding whether to ask the Public Platform team to build a new feature to allow this - unlike 3-vote close, there's nothing already built that will allow a change of this sort.

This question is the second part of that research - I asked a version of this question to the Moderators in November 2021 and am following up here to see what others think, too. Based on their initial feedback, they were concerned about a few things and it didn't seem there was a huge amount of interest in this path forward, at least as proposed at that time. That said, I've adjusted the description below to answer questions the moderators had and concerns they expressed as their arguments seemed reasonable.

What is the goal?

First things first - this project is purely information gathering as mentioned in the Q4 roadmap. We're trying to see whether it's something that would be worth building at some point in the future and we have no specific timeline related to actually building it. We'll be gathering data and community sentiment before deciding whether to move forward. Additionally, this wouldn't be shipped network-wide if we did build it. Like 3-vote close/reopen, it'd be up to the site to opt-in. That out of the way, let's look at the goal of the feature...

As my research showed, there's a need across the network to make curation tasks easier to complete and require fewer people to participate. Three vote close/reopen is a way to do that and it's also a very even-handed one - essentially, everyone with the privilege to vote to close/reopen gets the same decision weight when it comes to participating in reviewing and voting to close and reopen.

In general, this is fine, but it means that we're not necessarily recognizing the specialized knowledge and experience of people who are very involved in the curation process, and I would say that recognition is deserved!

So, in the end, the goal is: Make it easier for engaged curators to close and reopen questions more quickly, for any reason. Additionally, we hope that people who are working to curate the site will feel like their efforts are more valued and that there are fewer close/reopen review tasks needing their attention.

While we can't really hope to make reviewing fun, we can hope to make it feel less futile.

Why are we investigating weighting close votes as a solution?

Up to this point, weighting votes has been limited to the gold badge "duplicate hammer" - which, as its name implies, only has any power over duplicate closures.

There are other things about the gold badge hammer that may not be a good fit for this usage:

  • It's only particularly useful on sites that are huge (like SO) or have some very deep interest and expertise in specific tags (like SFF) since getting a gold tag badge is a very high bar to set.
  • Having a gold tag badge doesn't necessarily mean you know when to close/reopen posts - that said, having this only relate to duplicates minimizes that risk.
  • It only relates to duplicates.
  • It's unilateral. This is a pro and a con - for duplicates it's likely good, but for other close reasons, it may not be as sensible.

That said, when we initially rolled out this feature, half of the answer announcing it says that we'd consider it for other uses in the future - and that was in 2014! Seems like it's about time.

Beyond this, we think this is a good potential solution for the goal we're trying to achieve. While it's not something we can start implementing tomorrow, since we'd have to get it built into the site, we do have some data and personal experience that we can look at from duplicate closures.

What are the initial feature ideas for this?

I've got some ideas floating around in my head of what I think might make a good feature set to ask for here and that's based on my personal experience and my thoughts about the duplicate hammer. I'm open to being convinced that I'm wrong about any of these points! I'll put them here in a numbered list with my thoughts so that y'all can call them out by number in your answers:

  1. Increase the close/reopen vote weight to 2 for users who meet certain criteria - I think that making these unilateral has the risk of making it a bit too easy to close or reopen questions without a lot of oversight.
    • I could also go the route of making it a weight of 2 on sites with 3-vote close/reopen and 3 on sites with 5-vote close/reopen.
  2. Unlocked by voting to close/reopen (including reviews) n times + - Since reopening is less common, this would get imbalanced if we relied on each action individually. We don't necessarily want to rely on reviewer badges only as they're not very common. Review stats:
    • Steward badges (1k reviews) - only 88 sites have at least one person with the badge and only 42 have ≥10 †.
    • Reviewer badges (250 reviews) - 132 sites with at least 1 and 81 with ≥10 †.
  3. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.
    • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".
  4. Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.
  5. Applies to all close reasons - There's really no reason to restrict this to any specific close reason over the other. Except for duplicates, we don't necessarily want to incentivize people opting for one reason over the others.
    • This would not replace the gold badge hammer - so, no loss of privilege for those users when it comes to duplicate closures.

Those are the broad-stroke points. There are clearly minutiae, such as determining which close reason "wins" to poke at, but those things seem like secondary bits that we can work out once we answer the big question - "Is this idea worth pursuing?"

† - These badge counts are queue agnostic, so it's likely an overcount if someone has a Steward or Reviewer badge in LQP instead of Close or Reopen.

What do I need from y'all?

Poke at my ideas, question them. Point out things that I've missed or where I'm looking at things wrong.

  • Is this concept (even ignoring the specific rules I've outlined above) good/bad/silly?
  • Would this be something you think the sites you use would gain value from?
  • Is there something different that you think will address the goals we've mentioned?

Don't feel like you have to answer any or all of these questions. They're here as ideas for what sort of feedback I'm looking for.

17
  • 4
    Part 4, weighting votes in review, is not without precedent. It's done for 20k+ answer delete votes in the low quality queue: normally, answers must have a score of -1 or lower to vote to delete, but if voting from the queue, answers with 0 score can also be voted to delete. In other words, the privilege to cast delete votes on answers scoring 0 can only be exercised in the review queue. Feb 1 at 18:05
  • 4
    #5.... it probably shouldn't apply to "other"? other closures are questionable by design and should be seen by the full 3.
    – Kevin B
    Feb 1 at 18:24
  • 3
    Do you have ideas for how to handle retracted votes, especially retractions that occur after edits?
    – bobble
    Feb 1 at 18:27
  • 4
    @bobble I'm guessing they'd be excluded, either way. Sometimes it's a misclick, sometimes it's picking the wrong close reason, sometimes it's due to edits or even just realizing that you were wrong based on comments... I think the easiest option is to just pretend they didn't get cast in the first place.
    – Catija StaffMod
    Feb 1 at 18:38
  • 2
    @KevinB That's a reasonable point - I'll think about it. :) I'm guessing you're right but I'll need some time to cogitate. :D
    – Catija StaffMod
    Feb 1 at 18:48
  • 5
    @Catija, perhaps add the product-discovery tag?
    – Luuklag
    Feb 1 at 19:00
  • 6
    this past discussion looks related: Empowering tag-badge holders part II - let's look at silver?
    – gnat
    Feb 1 at 19:11
  • 3
    Perhaps a different topic (oft-raised over the years) but maybe "trusted curators" could be awarded some extra daily close votes. I think that would be popular on SO. (I used 40 of my 50 today in a single review queue!) Feb 1 at 20:45
  • 1
    There used to be this thing flag weight. Maybe worth recalling why it was abandoned. AKAIK just because it wasn't really doing anything. Feb 1 at 21:35
  • "3. ... "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good"." - That could just as easily result in all of the good reviewers not having any access to this while the bad reviewers do. If consensus in queues is consistently wrong because of robo-reviewers or because of lack of onboarding then the reviewers working correctly will be penalised for it. I like the idea, I don't know if automating who has these weighted votes is a good plan though. Feb 2 at 0:39
  • 1
    I also you seen meta.stackexchange.com/questions/240700/… Feb 2 at 9:54
  • Whatever rule(s) you decide on, allow someone with my history/content to hammer dupes & other poor posts.
    – philipxy
    Feb 2 at 10:43
  • 3
    Last year there was a moderator at SO who worked for many hours at a time to single-handedly close tens of thousands of questions, using special moderator abilities (the infinite voting weight). If I remember correctly, in the end, people thanked him more for what good he did than being critical of for example the quality of his reviewing activities. This could be seen as kind of a precedent for higher weights. One could even scrap (or strongly increase) the daily review vote limit (maybe multiply by 20 and make it a weekly limit in order to allow people to work more and more flexible).
    – Trilarion
    Feb 2 at 12:07
  • Regarding the topic dependence: when reviewing I mostly try to skip all questions where I'm not sure if they should be closed or remain open. I basically wish a suitable system would simply learn from my skips which tags/questions I can review well and which not, instead of specifying a list of them.
    – Trilarion
    Feb 2 at 12:14
  • 1
    The UI/UX team should be taking notes right now. Feb 2 at 21:40

20 Answers 20

53

Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.

Please don't do this. Some people close many, many more questions outside of review than in review. That should not deem their votes less valuable. Additionally, review queues on some sites are almost always empty (like here on MSE or on child-meta sites).

As an example, here are my current stats:

  • On Stack Overflow, I've submitted 3,110 close votes vs 102 reviews1 in the "Close votes" review queue.
  • On Meta Stack Overflow, I've submitted 140 close votes, only 8 of which are from review.

1 Only 80 of them are close votes.

7
  • 11
    Exactly my experience. Anecdotally, on my birth site (Puzzling) I'm often the one to start a close-vote train, which means others get to review.
    – bobble
    Feb 1 at 18:22
  • 3
    Some sites need more reviewing, some need more close vote chains started Feb 1 at 19:17
  • 3
    To give a bit of background on this - the focus of my 3-vote close/reopen project was to solve two problems - 1. votes aging out of review and 2. mods doing all the closing/reopening. In both cases, the need is to get more people reviewing posts. In the latter, you could argue that more people casting first votes would also be useful but it's often mostly the need for more reviews. As such, my original question when considering this project was "How can we get more people reviewing so that stuff doesn't age out or get left to the mods to handle?"
    – Catija StaffMod
    Feb 2 at 5:07
  • 3
    The side effect here is, yes... it may risk disincentivizing people from casting that first vote (or voting on the post directly). It's not so much my goal to do that - it's to help sites where they're struggling to get stuff closed at all (ages out of review) or needs to be closed more quickly (gets answered before getting closed). That said, as many people have pointed out, rightly, that's maybe a shortsighted solution - and so I made it an option rather than the primary feature... but the goal really still is to get more people to review.
    – Catija StaffMod
    Feb 2 at 5:10
  • 1
    @Catija Getting more people to review is a fair point indeed. I have two remarks though. 1) Voting on the post directly does not necessarily mean that it's the first vote; oftentimes, it's the second or third. Feb 2 at 11:07
  • 3
    2) Let's consider what happens if we increase the vote weight of users who mostly vote directly (ones that meet the criteria) vs not increasing it. If it's not increased, they will keep casting a single vote per post, pushing more and more posts into the queue (when it's the 1st vote) or keeping the post in the queue (when it's the 2nd vote). Now, if their vote weight is increased, in many cases, posts might not even need to be pushed to the queue, to begin with, when the votes are cast by this group of "trusted" users, which will indirectly help decrease the number of posts in the queue. Feb 2 at 11:07
  • 2
    I thought about posting this as an answer, but I'll just leave this here, since it's an argument against the review queue only option. Users can simply filter the close review queue to the tags on a given question to find its review task and vote from there with increased weight. Sure, it will stop some (most?) users, but for the motivated user who wants to exert undue influence, it's just a few steps away. Feb 4 at 1:34
23

Whatever the rules are, please make them as simple as possible

People are already confused by how sometimes moderators and gold-badge holders can unilaterally close/reopen questions. Fortunately, those are both quick to explain. But here, some of the ideas you're talking about, especially if we have them interact, throw in edge case logic, etc. etc. - all of a sudden it becomes a headache to understand. How are we going to be communicating all of this?

Sure, the idea of "trusted voters" might feel intuitive, but how it's implemented may very well not be. What a "good question" or a "good question record" is, is another example of a simple concept where people are constantly being confused by implementation. Whatever the rules are chosen to be, they will need to be clearly and simply presented, or we're going to have an explosion of poor confused users to deal with.

Please remember that simplicity is a virtue.

2
  • 7
    While I agree that it would be ideal to make the rules simple, that might not necessarily lead to the best results. I think a possible solution (if the "best" rules turn out not very intuitive) is to have a "Trusted voter" label next to the voter's name on the post notice, timeline, etc. (similar to how the gold tag badge is displayed currently) and have that label link to a FAQ question that explains "the rules", which we can also direct users at when they ask about this. Feb 1 at 18:47
  • @41686d6564 why introduce another label when the gold badge already does that? It's easier to implement and explain simply broadening the gold badge definition to all close reasons, and it would be seen as fair.
    – Braiam
    Feb 1 at 19:06
17

I like this idea primarily because it speaks to a belief that I think you and I both share: Tying user-level moderation abilities, and the weight they carry, to non-reputation validity more easily engages users who have a track record of making good user-level moderation decisions instead of engaging users who simply post upvotable content, and overall has the potential to improve their site-cleaning efforts (i.e. the net result is a cleaner site).

Making it easier for users who have demonstrated that they know what they're doing with moderation privileges to give back to the site in a more meaningful way is something I'd like to see more of. This methodology doesn't have to replace reputation (though that could be interesting...) but it makes users who spend a lot of their time flagging, voting to close/reopen/delete, and reviewing feel rewarded by further enabling them to keep fighting the good fight in keeping the site clean.


So... I like the idea of weighting close/reopen votes more heavily for users who demonstrate that they're performing these actions correctly... But how we determine who qualifies is going to matter a lot, and is likely going to be tricky. Your proposal...

  1. Unlocked by voting to close/reopen (including reviews) n times + - Since reopening is less common, this would get imbalanced if we relied on each action individually. We don't necessarily want to rely on reviewer badges only as they're not very common.

Having a minimum contribution for this is kinda necessary, so this is great. A concern you recognize is that not all sites have the same activity levels for a network-wide requirement for something like this, so some tweaking is probably required, but... Once you've helped vote to close/reopen enough, you should have a good idea of how it all works on the site you're helping to curate. If you haven't yet, your second criterion handles this:

  1. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.

    • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

This has a lot of potential to be good. While Sonic's answer raises concerns about cases where the validity of a closure isn't always correct, I think that the grand majority of question closures will reach a consensus that overall is correct. In other words, I feel like the cases that Sonic mentions would fall into a very deep minority, and Sonic, who is quite the prolific contributor here, would certainly meet the requirements to have his reopen vote weight increased under this system so that he can help correct mistakes like the ones he mentions more effectively.

What matters the most here, in my opinion, is the threshold that we choose for "valid" votes cast, in combination with ensuring that later edits to the post that result in it getting reopened do not mark the previous close votes as invalid. I don't want to get ahead of myself here and start suggesting good % thresholds for this, and I suggest that if we go this route we open up a discussion here on Meta to talk about what a good threshold should be. I expect there'll be a lot of squabbling but I imagine the end result should be something usable. Like you mentioned, some work is required to work out a good definition of "good."

Further, in case this isn't already being considered, how close a user is to this threshold should definitely be exposed to the user. It may not need to be exposed to other users (maybe moderators?) but how close you are to this vote weight increase should absolutely be visible in some way. I don't want to have to guess at how close I am to this vote weight increase - I'd like to be able to see it as something to strive for.

Overall, I think that this idea is heading in a generally good direction, and I do feel like this has merit enough to be built with refinement.

Running through your final feedback questions:

Is this concept (even ignoring the specific rules I've outlined above) good/bad/silly?

I feel like it's certainly worth exploring. I'm a bit demotivated to hear that this proposal's initial version which you proposed to the network's moderators resulted in little interest to pursue this further, because I do feel like, if implemented properly, this could be very beneficial in keeping the site clean and acts as a stepping stone towards giving the people who show that they're good at user-level moderation expanded tooling to help out more.

Would this be something you think the sites you use would gain value from?

I primarily frequent this site (Meta SE), Gaming.SE, and a few Beta sites. While Meta SE's barrage of blatantly off-topic questions never stops and requires frequent attention (that the MSE moderators, and yourself, are always helping with, thanks by the way!), Gaming.SE certainly has some very active curators who I feel would certainly love some additional close/reopen vote weight. The site garners quite a few Minecraft questions that aren't always up to snuff and might need closed so that they can be refined, or just deleted. These questions are typically honeypots for low-quality answers that also end up being deleted after the question is closed, leading to overall confused and demotivated users. Faster closure in those cases would be beneficial.

I'm very interested to see how (or even if) this idea moves forward.

5
  • 3
    The paragraph about the mods was poorly phrased and so I've updated it. The version here is significantly enough different that their opinions may be quite different. The post was positively received but there were concerns. In particular, the original proposal had item 4 as the default behavior. The feedback was clear that this was concerning, so I rethought things and realized my concerns could be mitigated in other ways.
    – Catija StaffMod
    Feb 1 at 21:45
  • 1
    @Catija Oh, awesome. Thank you! I feel much better that it was received in that way.
    – Spevacus
    Feb 1 at 22:06
  • 1
    "What matters the most here [...is...] ensuring that later edits to the post that result in it getting reopened do not mark the previous close votes as invalid." This line is doing a lot of heavy lifting, and is probably the hardest part to get right. Not done reading all the existing answers yet, but I plan on posting my own response that will include some thoughts on this danger.
    – TylerH
    Feb 1 at 22:59
  • 3
    Regarding your first paragraph - yes. When we gate by rep, anyone who can earn that rep can use the feature, whether they understand it or not - this means we must build in some amount of room for people using the privilege "wrong". If we adjust our process to make it so that privileges must be earned by active participation and use of a tool or going through a training/course/hula hoop, the need for extra checks on the privileges reduces. E.g. if we require someone to have a good suggested edit history before they can edit directly, we can grant direct editing earlier.
    – Catija StaffMod
    Feb 2 at 5:18
  • Regarding point 3, I needed to keep that paragraph brief because there are so many caveats and so much to decide about what a reasonable "good" is - and I trust that, if we move forward, we'll look into the data we have as well as discussing mitigations. We'll likely also make it pretty highly configurable so that we can adjust as needed to find a good target and review the actions that caused someone to gain/lose the privilege. We could probably even do a dry-run where we follow whether someone would have the privilege or not without actually granting it.
    – Catija StaffMod
    Feb 2 at 5:24
12
  1. Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.
    • "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

Something to consider is by using a percentage of "valid" votes there is a potential snowballing of the privilege. The fact that weighting votes has the potential to make someone with the higher weighted closure vote privileges more able to maintain the privilege.

Assuming the simplest possible "valid" close/reopen definitions: "the post being closed = valid closure" and "the post being reopened = valid reopen". Then someone with the ability to more easily close/reopen a question is likely to automatically gain a higher percentage of "valid" actions.

Within review queue it is fairly easy to see if there were mixed thoughts on a particular post i.e. some users voted to leave open and some voted to close, however, outside of review queue there isn't a way to dispute an "invalid" close vote until the question has been closed.

So my follow up questions would be:

  1. Should review tasks that have mixed leave open/close decisions be handled differently than those tasks with unanimous decisions?
  2. Is it an issue that someone with the privilege has an easier time maintaining it than someone trying to obtain it for the first time?
    • If it is how could this be mitigated?
5
  • 4
    I agree. One could specifically 'target' posts in the review queues that have 2 CVs (with the same reason) and just go along with those, whether or not they are actually correct. (The "pile them on" mentality I spoke of in my answer.) Feb 1 at 20:51
  • @AdrianMole How can one target posts that have 2 CVs? Or do you just mean "keep skipping until you get one that has 2 CVs"?
    – TylerH
    Feb 1 at 23:00
  • This is a great point Henry, and something that is extremely common in real life, unfortunately. For example, if everyone at a company gets a 5% raise this year, the person making $50,000 is now making $52,500, but the person making $100,000 is now making $105,000. They both are making more, and got the same % bonus, but the person who was ahead already is now further ahead... so much for an "equal" bonus. A "regressive" modifier would be great here, but would be fairly complex to devise and implement.
    – TylerH
    Feb 1 at 23:03
  • 1
    @TylerH Yes - keep skipping until... (But I suppose one could come up with a user-script that helps you do that?) Feb 1 at 23:04
  • 2
    1 - probably. And we should probably take into account the specific reason chosen for the close vote along with the edit history of the question. 2. In general, I want to trust that people will use the tools within community guidelines. In practice, I understand that's not always the case. I certainly do think that mitigations should be considered, whether that's flags for mods when people do seem to be gaming the system and/or ways for mods to manually revoke the privilege. We could probably also heavily downrank people who close questions that get reopened without edits.
    – Catija StaffMod
    Feb 2 at 5:30
9

Overall, I think this would be a good idea on those sites that have bloated Close Votes and/or Reopen Votes review queues, like Stack Overflow (which is the only site on which I have the CV privilege, as it happens).

But a few notes on some of your particular points …

There are other things about the gold badge hammer that may not be a good fit for this usage:

I agree. Linking this extra weight to tag-specific expertise doesn't make much sense, IMHO. While recognizing duplicates and providing the correct target(s) certainly requires specialized knowledge, closing for a number of other reasons does not (and cannot). For example, what of questions on SO that are off-topic because they are about general computing or network administration (amongst other things)? Who would have a gold tag-badge in a subject that is inherently off-topic? Also, some questions should be closed (IMHO) because they don't have any appropriate tags (such as a programming language).

Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.

Not so sure about this one. I know of a number of active users on SO who are very good curators but who aren't especially active in the formal review queues. Personally, I value their experience every bit as much as those who do 100+ reviews per day.

Requires maintaining n% of close/reopen votes & reviews deemed "valid" - Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.

This is important – robo-reviewers should not have this extra privilege! However, the exact mechanism for defining good (as you have already said) may be tricky. There are occasions when the Close Vote queue seems to attract "pile on more of the same" type votes that are actually wrong (either in the specific close reason or the actual decision to close). You have said, We'd put some work into finding a reasonable way to define "good". – that's likely to be quite a lot of work (but important).

And a couple of short questions:

  1. Would moderators be able to remove this privilege manually, if a user is seen to abuse it?
  2. Would the double-weight close vote still count as only one vote, in terms of our daily allowance. (I don't see any indication to the contrary in your post, but I just want to be sure.)
12
  • 5
    1. This is a concern that I have. I was considering that we might tie it to review suspensions at minimum. So a moderator could remove the ability by suspending them... they're not perfectly connected but it'd remove the perk and reviews without entirely removing the vote to close privilege. At some point we'd like to break from using rep for privileges in more ways but we haven't moved that way yet. We could also create a tool for mods to remove the privilege separately from using review suspensions. so we have options.
    – Catija StaffMod
    Feb 1 at 21:36
  • 5
    2. Yeah. It's still one vote. :)
    – Catija StaffMod
    Feb 1 at 21:39
  • I would think moderators would be just warn or suspend users like they do for other site abuse/infractions. At most, I think suspension from the review queue, specifically, would be plenty.
    – TylerH
    Feb 1 at 23:07
  • @TylerH I've already heard moderators concerns about abuse of the gold hammer: there's no way for them to remove that privilege other than a full-blown site suspension. Feb 1 at 23:08
  • 1
    @AdrianMole Sure, and I think that a full-blown site suspension is appropriate there; I wish moderators used it more often for gold hammer abuse.
    – TylerH
    Feb 1 at 23:08
  • "Linking this extra weight to tag-specific expertise doesn't make much sense" how so? If you are going to argue that, give specific examples as of why, not just "it doesn't make sense" because as far I know, closing as duplicate is still closing. So, it actually makes no sense that only duplicates has this behavior instead of all close reasons.
    – Braiam
    Feb 1 at 23:53
  • 1
    @Braiam I was under the impression that I had been specific: both for why tag-specific expertise is required for duplicate closure and why it is not appropriate for off-topic questions. But maybe I misread my post? Feb 2 at 0:18
  • @Braiam I am not here addressing (or even attempting to address) the issue of whether or not the gold hammer should be extended to all close reasons (and I am aware of your strong desire that it should be); rather, I am arguing that the proposed "half-hammer" (or 2/3 or 3/5 hammer) would be better based on criteria more general than specific tag expertise. Feb 2 at 1:00
  • 1
    @Braiam I'm slightly confused at your comment here - like Adrian, I see examples in this answer that make perfect sense. How can anyone be an expert in a tag that's off-topic? I also find it slightly amusing that you ask for supporting evidence here and yet your own answer has none of its own and doesn't even refute or even recognize the arguments in the question that clearly explain why I don't think the duplicate hammer is a good fit here.
    – Catija StaffMod
    Feb 2 at 5:33
  • "I am arguing that the proposed "half-hammer" would be better based on criteria more general than specific tag expertise" yet you don't explain why it's better: what is the indicator that would make this better compared to the flat behavior of the gold hammer? That is what I'm getting at @Catija, Adrian focuses in singing praises for it, but it doesn't answer the simple question: it's better at doing what and why should that be our yard stick?
    – Braiam
    Feb 2 at 11:18
  • Adrian offers some examples that are easily refutable @Catija, identifying non-programming question in a site doesn't need specific knowledge about the non-programming topic, only failing to identify the question as a programming is enough. (The question isn't "what is this?" but "is this programming?"). And that's not the only close reason that Adrian fails to correctly argue: in unclear I'm more than capable of knowing what an apt-get question lacks if I know about it, not if I'm not knowledgeable which I avoid to do in cases where I know I don't know enough about the topic.
    – Braiam
    Feb 2 at 11:23
  • Which caused me no lack of frustration due people unhelpfully closing as duplicate of stuff that will not help, at all. So, of course I want other reasons to have the gold badge behavior applied because I've seen several times how badly the community closes stuff it doesn't know about.
    – Braiam
    Feb 2 at 11:25
8

I'm not particularly a fan of making reviews a requirement of the feature, but I could see poor review history (or even a large percentage of actions being reversed) as a way of kicking someone out of the privilege.

Another thing to consider is the kinds of users at play here. There's the heavy moderation users, and the heavy answerer users. Their habits on site will vary, as will how often they reopen and/or close. If the requirements for obtaining this privilege past having a gold badge too heavily favor one or the other, this will shift the balance in a way that going from 5 votes to 3 votes didn't.

1
  • 9
    Just to add something regarding reviews - the vast majority of my 8k close votes on SO were not cast during reviews but instead as I was naturally finding questions. I also spend a lot of time looking at newly posted questions, so very often I'm one of the first to cast a close vote (which would then push that question into a review queue).
    – VLAZ
    Feb 1 at 20:47
8
  1. Alternative option: Only weight votes cast in review - We could make it so the privilege is only awarded for reviewing and would only weight votes cast in close/reopen reviews.

That's the very last thing you should do. Instead of giving more weight to random reviewers who might be completely unfamiliar with the topic, but happen to have some badge or another in an run-of-the-mill tag like "string" or "image", you should give more weight to the users who come across a post naturally. I think these users will not only be more familiar with the topic, but they will also care more about the topic.

In addition to increase the weight for those users, you could, for example, also make it possible to access the now review-queue-only actions like "vote to keep open" etc. directly from the question or add direct way to go from the post itself to the ongoing review.

6
  • While I certainly agree with this sentiment, it would be laughably easy to circumvent, since you can (and often should) open review items in another tab to get to the normal (read: the 'natural' question page view). Maybe "security through obscurity" is enough here and the site can tack on ?cb=1 or something to the URL when the question is clicked on from the home page or another question, but not when opened from review, and the feature only works if that URI fragment is included... Just thinking out loud a bit.
    – TylerH
    Feb 1 at 22:54
  • 2
    There isn't actually a "vote to keep open". It's just an option for a review of other folks' close votes. Three such from the review queue won't actually wipe out any existing close votes. Feb 1 at 23:14
  • 2
    @AdrianMole They won't wipe out existing close votes, but they would contribute to finish the review. This would remove the bias that voting to close is much more accessible than voting to leave open. Feb 1 at 23:26
  • 2
    @Tyler that method of circumvent does slow down revewer nicely, the primary issue is to protect against people trying to do reviews as quickly as possible. Feb 1 at 23:46
  • It's possible to filter reviews by tag, thus choosing what subject you want to review and not coming at subjects you are less familiar with, for those there is the Skip button; which is so forgotten when most needed.
    – Rob
    Feb 2 at 14:56
  • 1
    @Rob This can only be done for yourself, not to keep others from reviewing posts they know nothing about. Feb 2 at 15:03
8

the goal is: Make it easier for engaged curators to close and reopen questions more quickly... Is there something different that you think will address the goals we've mentioned?

I would suggest to address this from a bit different angle, namely - make it easier for users engaged in other ways help curators in closing and reopening.

The only attempt at step in this direction was a red dot indicator. I think it turned into a failure, and most likely reason is that meaning of indicator is too vague to be worthy, "somewhere there are some reviews that may somehow benefit if you take some action on them - do hundred clicks to find out". Even engaged curators may find it useless and distracting - and as for other users, they probably just learn to ignore it after giving it a try once or twice.

What I would like you to try instead is something that would require people just a single click from where they browse now right into a question that has only one vote left for closure (with non-positive score) or reopening (with non-negative score). This could be achieved by giving such questions some dedicated visuals / indication in question list views.

Questions in the need for that single close vote can be indicated as grayed - just like currently system indicates answers with solid negative score. Similarly, questions in the need for reopen vote can get bolder / darker visuals. This way, even non-curators, just active users browsing the site / tags their usual way, will have an opportunity to contribute to closes and reopens with minimal effort, "just in time".

(In case if you wonder why above I proposed to complicate indication criteria by extending it with score - this is because I would prefer to make it possible for people to impact it with regular votes up and down.)


Above was a network wide suggestion. Now, here are some site specific notes. I am going to focus only on sites I am active at. I'll start from smaller sites. I actively participate in three smaller sites and none of them seems to have substantial need in higher weight votes. This is possibly related to scale of these sites allowing to actively involve diamond mods in closing and reopening.

I am also active at Stack Overflow and in my experience, it suffers from a bunch of problems that are unlikely to be resolved by increasing vote weight. Specifically, your focus on trusted / engaged curators makes me feel that you ignore folks who are "merely" interested in maintaining quality of content only in their tag(s).

I am uncomfortable with that because I myself started curation maintaining content only in my tags. It went quite naturally: it was manageable because I focused on just a few tags and it was easy because the tags were familiar. Also back then it seemed to be in my own personal interest which motivated quite a bit (for the sake of completeness, later I decided that general curation is more in my interest but that's maybe a different story).

Nowadays, folks willing to maintain their tags seem to be in much worse position compared to how it was when I was doing this.

To start with, close queue became inefficient because it contains either "gray area" questions which are of high enough quality to avoid triage or fairly old poor questions that were released after triage (which additionally may be answered because of keeping open for a long time, which kind of trashes the very purpose of closing). And really poor new questions hang open for too long because of being held hostages in triage. It is quite likely that this results in folks dropping out of close queue after giving it a try once or twice.

This probably discourages people from using close queue (both engaged curators and folks who would want to maintain only their tags). I suggest to fix this issue by pushing poor quality questions into close queue in more timely manner (related feature request).

As for triage itself, it is unfortunately unusable for those willing to focus on curating specific tags. I've read somewhere that triage algorithm catches about 20% of total questions. This is great, and percent of abundantly poor questions seems about right, but this also means that if someone wanted to triage questions only in specific tag(s) they would have to make hundreds skips over questions in other tags, this is just unrealistic.

I suggest to give active tag users an opportunity to participate in triage by allowing to filter it by tags, just like it is currently done in close queue (related feature request).

The last but not the least issue is huge amount of close-worthy questions in popular "big" tags. These tags are flooded by hundreds or even thousands inappropriate questions a day, which makes it doubtful that fairly narrow community of trusted curators will be able to handle such flood, no matter how much power you give to them.

Issues of this scale would better be handled by measures of appropriate scale. Specifically, I think you can try to leverage the fact that "big" tags tend to have respectively large numbers of tag badge holders and give these folks more close votes to cast in their tags (related feature request).

These folks are active in their tags anyway and it is easier for them to evaluate questions quality because they are subject matter experts and some (hopefully many) of them may be interested in stopping flood of homework dumps that pollute big tags and obscure answer-worthy questions.

2
  • 3
    as a side note - given amount and severity of issues found in recent UI changes I think it would be better to introduce stuff like special indication for questions missing a single vote gradually - say, make it visible first to only 10K users, then to 3K etc
    – gnat
    Feb 5 at 8:13
  • 1
    I like the idea of trying to engage more people as the window to act nears its end rather than giving a handful of people more power. For healthy communities, increasing engagement seems better than decreasing the number of people needed to make a decision.
    – ColleenV
    Mar 21 at 16:38
6

Weighted votes are problematic in some aspects and will not solve core issues.

I am talking from a Stack Overflow perspective, since I have very little experience on other sites.

My first criticism would be putting close votes and reopen votes into the same category.

On Stack Overflow there are a huge number of questions in the Close Vote Queue, while the Reopen Vote Queue is small and manageable.

Reopen Votes

Currently on Stack Overflow the number of votes needed to either close or reopen is 3. For the reopen queue this is a rather low number, because high reputation users can vote to reopen their own questions. That means only two additional votes are needed for reopening which is a rather low bar.

Also I have seen plenty of blatantly off-topic (totally missed the site) questions having two reopen votes and plenty of those even get reopened. It seems like fresh reviewers with lower reputation prefer to reopen questions (regardless of whether the question is on-topic or not) rather than keep them closed, which is another problem.

We also have occasional close/reopen wars where even gold badge holders are engaged and are abusing their power in order to gain more reputation, so they reopen obvious duplicates. Again reducing the number of votes needed to reopen questions that need to be closed would only increase potential abuse, because even more questions that don't deserve reopening would be reopened.

If some question really deserves to be reopened and there is a problem achieving that, we have the SOCVR chat room where people can ask for a question to be reopened if they think it is in good shape.

We don't have a problem with reopening questions; we have problem with closing. Any fiddling with reopening votes in sense of making it easier to reopen questions would be bad for the Stack Overflow.

Close Votes

Three votes to close question have significantly increased our ability to successfully close questions. I think three close votes are optimal for closing non duplicates, as it is not too high, but it still requires some consensus from multiple users.

Why weighted votes are not a good option has multiple aspects.

First, the main problem on Stack Overflow is not the number of votes needed to close question, but the sheer number of questions that need closing and not enough people willing to moderate the site. Weighted votes will only make a small dent and will not bring us closer to solving the influx of poor questions.

That problem can only be solved from multiple sides, including better education of new users about rules and guidance on on-topic and off-topic questions.

Weighed votes would increase frustration for new users and increase tension, because even now users complain about their questions being unilaterally closed when it comes to duplicates where the OP is pointed to another question where they can find the answer.

Closing question by single or even just two users, for seemingly on-topic, but otherwise poor or unsuitable questions, would significantly increase the friction and accusations toward close voters and we would still not be able to close all that need to be closed.

Besides questions that are just poor questions, but might be edited to shape, there is still a huge amount of blatantly off-topic questions asked and questions asked in other languages (not English).

Being able to close those questions with single vote would be extremely useful. But, weighted votes are not perfect here as they either require a gold badge holder which makes no sense for blatantly off-topic questions as you don't need to be an expert, or two people which is still not fast enough given that there is just too many of such questions.

Also, some of such off-topic questions can be migrated or closed as belonging to another site, but only for a limited number of target sites, but others fall into unclear (non English) questions or don't belong anywhere on SE.

To allow faster closing of such questions it would be far better to have a completely separate blatantly off topic or unsuitable reason that could be paired with another review queue, so that such questions still go through some additional vetting (preventing power abuse) while we would have ability to cast a single vote to close them as fast as possible.


If you do opt for weighting votes, please don't limit those to review queues as plenty of people close questions outside the queue. For instance, I have cast 36,785 close votes on SO and of those only 6007 from the review queue.

Additional help for solving closing problem would be prolonging vote aging periods and giving more close votes (there were days when I would easily bump into 40 votes limit, and could cast more if I had) to people that do moderate the site.

0
5

Here is me poking at the idea.

I see where it is coming from. It would make reviewing even more efficient, i.e. result in more reviews done with the same investment of human resources, albeit with maybe a higher error rate. Even domain experts make mistakes from time to time (and sometimes even are biased). The idea of the 3 close votes requirements and the 5 close votes requirement before that was that we wanted consensus to reduce the overall error rate. With an increased vote weight as proposed here we would have much less of a consensus effect than before.

And indeed we not only need assurances about the decision to close or not to close but also about a suitable close reason. The close reason is the most important information for the content creator, so it better be right. How do we know that we present the right close reason(s)?

I agree with the general idea (to weigh review votes somehow by trust/experience), but I see the same problem I saw when we reduced from 5 to 3 votes and will repeat it here:

We must control the error rate of reviews

Why having a vote weight of 2 for experienced users, why not 1.64 or 2.47 and a threshold of Pi or something else completely arbitrary? Obviously there should be arguments why 2 is the best and nothing else is better. The argument should be that at this weight the error rate is still reasonably low while the reviewing efficiency is already quite high.

Ideally you would measure the error rate (by letting multiple people do the same review independent from each other) and then compute a review efficiency vs. error rate relationship in dependence of experience and select an experience dependent weight that controls the error rate uniformly over all experience levels from there.

Measuring the error rate (if we assume that reviewers get it right on average and aren't biased on average) might be possible (also adding a few more false positives in the data set). This idea here only makes sense if the error rate indeed goes down with increasing experience. However, if this is the case, we might not need experience as proxy for the error rate, simply estimate an error rate for every single reviewer and take this as inverse weight. A new user could be a better reviewer than any gold badge holder. Why not? Just measure directly what you want to know.

I could imagine a system where everyone starts with weight 1 (or even zero for really new users) and depending on his review actions can either increase or decrease his weight.

And finally one more thing that needs attention:

Reduction of close-worthy content could reduce the need to review that much

Sure, there will always be content that needs to be closed and everybody makes mistakes, but a lot of duplicates might be avoided if only people searched more and a lot of unclear, unfocused questions or questions without debugging details (on SO) might not need to be closed if only the content creator would have been more careful and more knowledgeable when creating the content.

The amount of reviews and closures necessary hints that there are quite large problems with content creation that happen much before reviewing takes place.

While better education, guidance and motivation of content creators could be seen as complementary to this idea here, it affects the urgency of it and therefore should be kept in mind and resources should be spent on it equally.

This is strongly related to:

While we can't really hope to make reviewing fun, we can hope to make it feel less futile.

Reviewing can be fun (it's nothing else than giving feedback on existing content in order to improve it), if only the reviewers get the impression that the content creators invest the same amount of time and concern in the creation and improvement of existing content than the reviewers themselves invest into reviewing. If only that would be the case.

4

We don't really want to incentivize people to use duplicates when other reasons apply. I would love that all close reasons behave like duplicates close reasons. If I can be trusted to know that two questions are the same, I also can be trusted to know that a question is unclear, is not within the site defined topics, is too broad or opinion based.

I don't really want the system to be more complex than it already is, I want to know that if I cast my vote I know exactly what will happen: either the question is closed or my vote goes towards the pool of votes. Being surprised that my close votes behave differently in different context makes zero sense to me. If I cast a vote, wherever I cast it I want it to behave the same way.

13
  • 4
    "If I can be trusted to know that two questions are the same, I also can be trusted to know that a question is unclear, is not withing the site defined topics, is too broad or opinion based." Maybe you can be trusted with that. But many other gold tag badge holders couldn't really be trusted to determine whether or not something is on-topic. I could give examples but I don't want to call out specific users. I will just say that I've seen several discussions on meta about users with very high reputation who hardly know anything about the site rules. Feb 1 at 19:20
  • @41686d6564 and you will prevent the 99.99999% of the rest of the users to do that for a group of users? Better to have them doing exactly that so that they can be educated when they get it wrong, than be forever in ignorance.
    – Braiam
    Feb 1 at 19:24
  • 12
    Hmm. While this may be true, I still think it's not a great idea just because a gold tag badge relies on having answered questions, while closing questions means they can't get any answers. There's a bit of a "conflict of interest" between the two, I'd rather see more close voting power tied to actual things related to close voting than to answering.
    – Tinkeringbell Mod
    Feb 1 at 19:25
  • 2
    @Braiam While I don't agree with everything in Sonic's answer, he brings up a good point: "using the same criteria of gold tag badges (or tag badges in general) is likely not a good idea since the tag badges only count one's skill in answering questions". I think that if the criteria involve the success rate of one's close votes (as proposed in the question), that would already include the number of gold badge holders who have demonstrated their skills in closing questions, but exclude those who did not. Feb 1 at 19:36
  • @41686d6564 no, that doesn't count as skill, that count as a answers that others found worthy of upvotes. If you want to reign them in, just delete the answers or the questions. Remember, everything else apply, they can't reopen a question more than once, so removing the sub-par contribution is a solution.
    – Braiam
    Feb 1 at 21:56
  • @Tinkeringbell what conflict of interest? How does it benefits me that I can close questions and that I seem to be posting answers that others upvote? Also, it's not like they have unlimited power either, they get one shot, like everyone else. If other users vote the other way, it can only overrule them once. As someone that has gold badge, I find myself wanting to close questions with other reason that are not duplicate.
    – Braiam
    Feb 1 at 21:59
  • 1
    @Braiam I put the conflict thing in quotes because I'm not sure what else to call it... A tension, perhaps? But you see the same happening with tying privileges to reputation... It may work to prevent abuse by some, but having the required reputation to use a privilege doesn't mean a user knows what they're doing, and plenty of them still end up e.g. robo-reviewing. So I want to see the more powerful version of close-voting tied to using your less powerful close/reopen votes correctly and not tied to having a certain score for your amswers in a certain tag for that reason.
    – Tinkeringbell Mod
    Feb 1 at 22:10
  • 2
    Also, if you let gold tag badge holders single-handedly close or reopen question with a certain tag for any reason, I fear the accusations of "you're preventing others from getting the same power as you" will start flying soon...
    – Tinkeringbell Mod
    Feb 1 at 22:14
  • See from where I'm sitting at @Tinkeringbell, on one hand you claim that the gold badge close votes has been a resounding success, then you put some conflict of interest on the other for doing the same thing (remember, closing as duplicate still prevents new answer to be posted), so which is it? We can't have it both ways, and you seem to have forgotten that SE is trying a data driven approach, and data says "it works!" So, how about stopping with the unfounded fears of unknown ghosts and gather actual data instead? Who knows, maybe it was actually a bad idea.
    – Braiam
    Feb 1 at 23:50
  • 4
    We don't really want to incentivize people to use duplicates when other reasons apply. - We already do... for gold tag badge holders. This would technically reduce the incentive to duplicate close something because it weights all of the close reasons and still leaves gold badges as a perk for extremely active people in a specific tag. Plus, to some extent, I'd much rather someone close something as a duplicate (that is one) than close it with no duplicate because the duplicate at least gets the asker on the road to an answer while the other close reasons just shut the door on them.
    – Catija StaffMod
    Feb 2 at 5:40
  • 1
    People have already requested that we remind them when their duplicate hammer is in effect - so it's already confusing to some people. As such, granting people 2X votes to close/reopen across the board is exceedingly simpler than someone having to remember which tags they have a gold badge in. While I understand what you're trying to say, I'm sorry - I think you've got things quite backwards. :)
    – Catija StaffMod
    Feb 2 at 5:43
  • @Catija "I'd much rather someone close something as a duplicate than close it with no duplicate because the duplicate at least gets the asker on the road to an answer while the other close reasons just shut the door on them" and that's the fundamental issue. Correct duplicate closure is something that SE has identified as problem in the past, having all options would remove any incentive that close voters have to close as duplicate.
    – Braiam
    Feb 2 at 11:47
  • @Catija about confusion, I still can't see it. If you are not sure it's a duplicate, you shouldn't be voting at all, isn't it the correct behavior. I'm confused why that would be a problem? I expect that when I vote to close a question, it gets closed (eventually).
    – Braiam
    Feb 2 at 11:49
4

there's a need across the network to make curation tasks easier to complete and require fewer people to participate.

Giving some individuals more power over which questions get closed or reopened is one way to do that. Another way to get more done with less manual labor is to automate some of the work.

There is going to have to be some effort to determine how someone's vote will be weighted more than someone else's and by how much, so maybe some of that work should be to determine if there are ways to off-load some work to an automated system. For example, (and this is just an example to explain what I mean, not something that I want to debate the details of) if a post has certain characteristics that are indicators of low quality posts, maybe one close vote by anyone is enough to close it, but it automatically goes into the reopen queue for confirmation. Quickly closing a low quality question prevents well-intentioned bear feeding and the reopen process tends to be more amenable to salvaging questions than the close process.

I think it is a mistake to weight close votes solely by who is casting the vote and their close-voting expertise, and not by the nature of the content they're voting on. The reason some sites still need 5 close voters to close a question isn't because the people reviewing are somehow untrustworthy or incompetent, it's because the content is harder to definitively and objectively separate into off-topic and on-topic and the queues aren't getting so overwhelmed that we need to reduce the quality of the reviews to keep up.

4

Thanks for posting this! I have quite a lot of thoughts... no guarantee they're all included here in the first pass:

There are other things about the gold badge hammer that may not be a good fit for this usage:

It would be good to revisit the topic of expanding gold badge powers as its own topic, but since you brought it up as a comparative example applied to this proposed feature, here are my thoughts:

  • It's only particularly useful on sites that are huge (like SO) or have some very deep interest and expertise in specific tags (like SFF) since getting a gold tag badge is a very high bar to set.

You mentioned above this wouldn't apply to all sites, only those who want it. How many would want it? We need some data on that, so we can see if there are actually any sites that would want it but would have trouble getting it. If so, that can be part of the "improve the gold badge holder close feature set" discussion... doesn't seem particularly relevant here.

  • Having a gold tag badge doesn't necessarily mean you know when to close/reopen posts - that said, having this only relate to duplicates minimizes that risk.

While true, there are other, non-mutually exclusive alternatives between the two extremes of "everything" and "only duplicates". Wouldn't a user who is considered a "subject matter expert" in a tag know if a question using that tag is, say, an opinion-based one that can't be objectively answered? The "opinion-based" reason seems like quite safe low-hanging fruit in terms of expanding the privilege. Why not start there?

  • It only relates to duplicates.

This is only so because Stack Overflow made it so. You could easily apply it to all close votes, I suspect. Certainly if you're applying the powers to another set of features, some changes would be not only opportune but also appropriate!

  • It's unilateral. This is a pro and a con - for duplicates it's likely good, but for other close reasons, it may not be as sensible.

Again, this can easily be changed.


  1. Requires maintaining n% of close/reopen votes & reviews deemed "valid"

While I agree that preventing robo-reviewers from having additional, more powerful abilities is ideal, I think we need more data (on users' vote reasons and how they tend to spread their votes across reasons, especially in the review queues) before we can discuss this to a significantly deeper level. In the review queues, I tend to spend most of my time narrowed by close reason or by topic. Some close reasons apply to questions that are permanently or inherently off-topic... a request for recommendations on which language to use for a project is never going to be on-topic. While it's not likely to happen, it does sometimes occur to OPs of such questions to delete the entire question and rewrite it as a new, totally unrelated one... which might well be on-topic.

Likewise, and far more important here, are the close reasons that are inherently temporal in their application: the "needs detail", "lacks MRE", and "needs focus" reasons (or as I like to call them, "Unclear", "No MCVE", and "Too Broad" reasons... seriously, could we get those old naming/descriptions back?). These close reasons can easily be invalidated (which is great!) by an OP who diligently edits their question to narrow the scope of it, add missing detail, or include a necessary error message/stack trace.

Such "how it is now"-dependent close votes are correctly cast in the moment, and should not be discouraged. Yet, if we start to pin future, powerful abilities on surpassing a certain threshold of "successful" votes... you might punish users who've cast these votes, or worse, discourage users from casting votes of a certain type for fear of shooting themselves in the foot.

I see two ways, of the top of my head, to help combat that problem:

  • Only count vote success for questions where the question has not been edited (by someone other than the voter) since the vote was cast.

And/Or

  • Only count vote success for questions that have been successfully/completely voted upon (e.g. for close votes, the question was either closed at some point or left open via review completion or all close votes aging away; same appropriate adjustments for reopen votes) and some time-based constraint such as:

    • where the vote was cast in the last 365 days.
    • the above, but vice versa (before the last 365 days, instead of since 365 days ago).
    • where the last status change of the post (closed, reopened, edited by user other than the voter) was more than 365 days ago.
    • the above, but vice versa again

(I provide conflicting options here because I've not thought enough yet about which one would be better)


Unlocked by voting to close/reopen (including reviews) n times +

I think this is a good metric, but it doesn't have to be static. I would tend to think more reviews are better, but 'age/seniority' is a stale metric by itself, in my opinion. One alternative would be to count "reviewing/voting to close or reopen n times or more in the last 365 days.

This would be unique in that it would be a time-based requirement that could be lost and regained, based on your reviewing activity. (E.g. "Haven't performed close/reopen moderation in years? Well, you'll need to get 200 (for example) close/reopen votes under your belt this year before you have it again" ... which is only 4-5 full days of voting or reviewing). This could be implemented via a sliding time requirement or factor in whether someone had it at the end of a year to grandfather them in at the beginning of the next year, to avoid people who have it suddenly losing it come New Year's Day.

Of course, it could be combined with a flat requirement of n close/reopen votes altogether, to ensure a minimum level of experience/exposure to the process and privilege overall.

This may be too complicated, or lead to too much confusion ("hey, I had double-weighted close votes in the past, but don't have them anymore!"), but I think it's worth considering. It would certainly help ensure accuracy by ensuring recent familiarity with the process... since content standards change over time on a given site (what was OK to ask in 2009 is often not OK to ask in 2022).


We don't necessarily want to rely on reviewer badges only as they're not very common.

On that subject, this is again problem a topic best discussed separately, but why not visit the idea of weighting badge requirements based on site activity, like you already do for Beta vs non-Beta sites? Either total questions, or questions per day... start out with lower thresholds for recently-graduated sites that are above Beta but below what Stack Overflow (for example) has, and once a site hits the threshold(s) the first time, it changes to the full/highest threshold and stays there permanently (to avoid it constantly changing back and forth for sites who hit a questions/day metric one day and then don't the next, etc.).

Only weight votes cast in review

As others have said, I think this is bad; many, many user moderation actions (most, even?) are done outside the review queues. Rewarding only users who perform them from a /review/ URL is unfair and would miss out on a huge chunk of users actually using the feature.

18
  • 2
    You have considerable review experience on SO but little on other sites, little experience and number of reviews. Throughout this complex argument you call for data but offer few opinions to repair the points you raised, also without data. --- It's not clear that this offers a solution for SO, or the more than 100 sites that could benefit. --- Yes, I read through this more than once; and the staff will have to too.
    – Rob
    Feb 2 at 1:07
  • 3
    Regarding point 3 - I did state that we'd need to do a lot of work to define "good". Things like considering whether the post was edited are absolutely on the list of considerations but I didn't want to take the time to enumerate them in the question because that's really a discussion for another time. What I need to know now isn't the minutiae, it's whether anyone actually thinks this would solve any problems, regardless of the specifics. :) There's no reason to "polish a turd", if you will. :P
    – Catija StaffMod
    Feb 2 at 5:55
  • 2
    Regarding the duplicate stuff - I think we're coming at this from opposite directions. You're trying to enhance/adjust the gold badge and give gold badge holders more privileges based on their SME status... and I'm trying to give everyone more privileges based on whether they've shown that they know when to close a question. I'd argue that many of the close reasons don't need an SME to identify what's too broad or lacking in detail. Why should we limit who can get those questions closed more quickly to gold badge holders only?
    – Catija StaffMod
    Feb 2 at 5:58
  • 1
    The thing is, on SO it's maybe more cut-and-dry... if you're not knowledgeable in Python, you might think a question is lacking in detail to be answered while a Python expert can answer easily and so, to some extent, it may be more beneficial for an SME to judge... but I'd argue that, even then, the bulk of questions that should just be closed quickly are pretty cut-and-dry and don't require expertise. But on sites like Cooking, you probably don't need to be an SME in baking to recognize when a question about eggs is lacking detail. Restricting it to badges really isn't necessary there.
    – Catija StaffMod
    Feb 2 at 6:02
  • 2
    I think this is a good metric, but it doesn't have to be static. - yes. I think I mentioned this in comments on bad_coder's answer.
    – Catija StaffMod
    Feb 2 at 6:05
  • 1
    @Rob Experience with the review system transfers equally well to all sites on the network. The only thing that changes between sites is the topic: what kinds of questions are on-topic for that site. I don't use review queues or cast close votes on other sites because I have not earned the reputation to do so, which is quite common for users on the network. If I had 3k+ reputation on other sites, I'm sure I'd vote to close often on those sites, too. I'm not sure what you are mean by "repair the points you raised", so I can't respond to that.
    – TylerH
    Feb 3 at 20:34
  • 2
    @Catija I agree most of the close reasons don't need an SME to know fairly conclusively, for the average question that shows up in the review queue. I'm not suggesting we should limit who can get those questions closed more quickly to only SMEs; my answer on that subject is merely suggesting we also allow SMEs to unilaterally close for more reasons (since your question raised that point/discussion). Your post above seems to indicate we don't use the SME model/implementation as a model for this because it's only for dupes. My point is that the SME model doesn't have to be only for dupes.
    – TylerH
    Feb 3 at 20:38
  • 1
    No, I think my goal is in saying that granting SMEs unilateral close votes for all reasons is problematic because being an SME doesn't mean you know when something should be closed. A good chunk of my question is involved with addressing how we avoid people who abuse the usage of this tool. So, the point being, I'd much rather have a level playing field (SMEs must qualify in the same way as a non-SME) than have a two-tiered system.
    – Catija StaffMod
    Feb 3 at 20:55
  • 2
    @Catija I also agree granting SMEs unilateral votes for all reasons is unnecessary, and possibly problematic; I only suggested granting them unilateral votes in a couple other reasons, such as "opinion-based". An SME should know even better than others when a question in their given language is going to be an matter of opinion, for example. However, I'm OK with giving a little extra weight to all vote types based on other criteria such as "experience in the user moderation world" so long as SME/Mjolnir powers are not hamstrung (though, I do have a thought about an improvement there).
    – TylerH
    Feb 3 at 21:01
  • That not true, people who use more than one site know that the close reasons have little overlap and vastly different levels and kinds of expertise requirements placed upon the reviewers; much like reputation review skill is not transferable. --- 3K reputation is neither required to review or to know that.
    – Rob
    Feb 3 at 21:35
  • 1
    @Rob The screenshots you are showing all discuss differences between ...what is considered on-topic on those sites... which is exactly what I said the differences are. So it sounds like you're agreeing with me. Also FYI I was referring to close/reopen reviews exclusively when I mentioned 3k reputation. Most sites have such little activity that the lower-rep review queues rarely have items in them for more than an hour or two.
    – TylerH
    Feb 3 at 22:24
  • You only need 15 reputation to be the first reviewer, and otherwise 3,000 or more reputation (500 or more on public beta and non-designed sites, 15 on private beta sites). I'm not saying you agree with me, because I understand what's being said and how reviews work; on more than one site. --- Quite a few sites, with their low activity, have fewer than a total of half a dozen reviews done over the last decade (the History doesn't show up full of avatars), and reviews sit for so long that they age away. --- Research!
    – Rob
    Feb 3 at 23:22
  • 1
    @Rob flagging is not reviewing. If you think it is, you may not understand how reviews work, after all.
    – TylerH
    Feb 4 at 15:53
  • "why not visit the idea of weighting badge requirements based on site activity" - I'd go one step further and suggest that badge requirements should scale based on the volume of questions in that tag, rather than across the entire site. Feb 13 at 20:43
  • @NotTheDr01ds I doubt that would be feasible, not least because it's inherently unfair. For example, in that system, I could unilaterally close a question because I have answered, say, 50 of the 100 questions with tag X on the site, but you can't do the same even if you have answered 150 or 200 of the 50,000 questions with tag Y on the site? In this scenario you have clearly contributed significantly more and 'proven' your subject-matter expertise more than me, but you have fewer privileges simply because the tag you're answering in is far more popular. That's not to mention gold badge wars...
    – TylerH
    Feb 14 at 17:31
3
  1. Requires maintaining n% of close/reopen (...)

    "Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

If we're talking about the scale of Stack Overflow the overall number of successful close votes should outweigh the usually small percentage of reopened posts. A lot of closures fall into routine maintenance so lets say a user that cast x number of successful close votes (somewhere in the several thousands) can be trusted to close their daily quota at increased weight because a bulk of the daily posts are straightforward to evaluate.

In other words, focusing on percentage only overlooks the fact that a close voter with 10k CVs at 95% remaining closed is more experienced and effective than a close voter with 1k CVs at 99%. (The 10k reviewer is 10X more effective at only a marginal 400 post difference to a 9000 reviewed post gain.)

EDIT:

I'm going to include two comments because I think they are relevant and I want to raise a use case as counterargument:

by Catija

I'd likely be considering recent participation, so either votes cast in the last n days or the most recent n votes rather than looking at someone's all-time stats.

and

by Akixkisu

They will also have no issue quickly regaining that trusted user status — since they can do their daily quota every day

The use case that comes to mind as a counter-example is the tag expert who's only engaged in one or two tags and tends to them daily (maybe without having achieved a gold badge). I see a lot of those CVs aging away and with the short sliding window those SMEs are likely to not see any change to their CV weight.

11
  • 8
    Yeah, I should have been more clear, I'd likely be considering recent participation, so either votes cast in the last n days or the most recent n votes rather than looking at someone's all-time stats. A good catch, though. :)
    – Catija StaffMod
    Feb 1 at 18:40
  • @Catija I have this reservation about short windows because very engaged reviewers will want to take the occasional break without loosing their streak. (It's not like the reviewer will forget his skills if he's too busy at work to review for 3 months...)
    – bad_coder
    Feb 1 at 18:42
  • 4
    I understand but purely volume-based metrics have the opposite issue - someone who takes 2 years off may not be in touch with what's currently close/reopen worthy any more.
    – Catija StaffMod
    Feb 1 at 18:45
  • @Catija I think the most experienced reviewers (whose trackrecord took years of contributions) would argue its like riding a bike - you never loose the hang of it.
    – bad_coder
    Feb 1 at 18:51
  • 8
    Policy can change, though.
    – Catija StaffMod
    Feb 1 at 18:53
  • "its like riding a bike - you never loose the hang of it" Are you sure about that? :-P Feb 1 at 19:06
  • 1
    A frequent situation that we have on rpg.se are longer-term users clashing with newer power users about how to handle specific issues; a lot of what these long term users experienced in the type of question and policy they navigate ties closely to the different environment at their active time — some of it translates 1 to 1, but a significant portion about their approach doesn't fit the current environment.
    – Akixkisu
    Feb 1 at 19:18
  • @Akixkisu a lot of the stuff on Stack Overflow is invariant, if the question is a duplicate or doesn't have code, etc... There are some types of close reasons that haven't changed since the site's beginning and I'm guessing those are the bulk of the problem. (You also realize the visible rpg.se corpus is equivalent to 22 days of SO?)
    – bad_coder
    Feb 1 at 19:21
  • 3
    @bad_coder Yes, we are here to talk about the network and SO is certainly an abnormal outlier.
    – Akixkisu
    Feb 1 at 19:26
  • 1
    @Akixkisu let me just strike this key once more (and I'm all for debating SE at large) there are some users on SO who closed the entire rpg.se corpus by volume, and some did it twice! (Are we really not going to weight in those achievements?)
    – bad_coder
    Feb 1 at 19:28
  • 3
    @bad_coder A user like that will have no issue to continue onwards as a trusted user if they remain active. They will also have no issue quickly regaining that trusted user status — since they can do their daily quota every day.
    – Akixkisu
    Feb 1 at 19:31
3

One option is to give "2 vote power" to both close and "don't close" to Silver/Gold tag badge holders so people can look after the tabs they monitor new questions on. I think this should only be done outside of review queues, as people tend to read questions more carefully when deciding if they will answer the question.

(This does include enabling a silver/gold badge holder to vote to not close before the question has been closed.)

Maybe a "leave open" review response should convert such a close vote into a single powered close vote, so the additional power only effects the most clear-cut cases.

4
  • 3
    I'm intrigued by your last paragraph as an option. It seems like it might be a way to balance some of the concerns I've seen but I'm not sure whether I agree with your statement that people who see questions outside of review read them more carefully. While there are certainly people who are robo-reviewers or who don't read particularly closely, I think there's also people who see lots of new questions in their tags and just assume they're the same as other questions and vote to close them... I think it's a problem in both places.
    – Catija StaffMod
    Feb 2 at 5:50
  • @catija Do you care more about advoiding being seen as unfriendly by new users or advoiding driving away the experts by showing them low quality questions? Provided great questions don't get closed, I have no issues with many "just about OK" questions getting closed in error. The tab badge holders are the experts, eg your important users. Feb 2 at 9:50
  • 3
    That... feels like a false choice. Questions that don't meet the site's guidelines should be closed. It's not about making any group happy. A big part of why closing is seen as unfriendly is a separate issue. The focus of this project is getting questions that should be closed and that valued, experienced close voters judge to be close-worthy... closed... along with making reopening similarly easy in cases where it's deemed appropriate. I don't think the tag badge holders are necessarily experts. What tag badge do you need to have to identify a shopping question?
    – Catija StaffMod
    Feb 2 at 14:58
  • @catija I think question that indicates too low a level of expertise/skill to understand an answer should also be closed. But it was decided we MUST welcome people who are not skilled enough to be worth having. Feb 2 at 16:17
2

Catija, you may remember me (or perhaps not) as having recently requested three-vote-close on Ask Ubuntu. I've since changed my mind on this, after having hit 3k rep myself there and seeing the (IMHO) poor quality of close-vote reviews there.

Unlocked by voting to close/reopen (including reviews)

Two problems with this:

  • First, the number of reviews someone has done does not make them an expert in all topics on a particular site. Even gold badge expertise isn't a good indicator. The knowledge needed to judge (and answer) a question is often not based on "general knowledge" in a subject area, but specialized knowledge in the topic being discussed.

    For example, a gold-badge holder in the "Windows" tag on Super User may (but more likely, may not) have expertise in (to take a random example), Gimp. A Windows Gimp question may even appear unclear to them, but perfectly clear to someone strongly familiar with that particular application.

    The same goes for someone who has voted on 1000 posts across the site. That should give their review of a Gimp question no more (or less) weight than anyone else.

    And this doesn't seem to be a "corner-case" to me. I very often find "Needs clarity" close-votes on posts that are perfectly clear to me (or at least salvageable by edit).

  • The second problem is that this just makes people rush through reviews even faster, further reducing the quality. For instance, voting to close because a "quick read" of something is "needs details or clarity" is far easier than editing it to make it more clear and readable. I rarely hit my review cap during any given day primarily because I spend more time editing (and sometimes even answering) posts there that others have been VTC'ing as "Needs details or clarity."

    Encouraging reviewers to "rush" their reviews in order to reach "trusted reviewer" status more quickly clearly has the side-effect of reducing the time spent and (by extension) the quality of those reviews.

Whether someone's reviews were actually "good" and whether someone should be able to robo-review and still get this privilege is a concern to me. I don't think they should.

I agree; I just don't think there's any reasonable way to do this algorithmically. No matter what you attempt to measure, it's likely to be gamed.

"Good" defined by being validated by other community members (e.g. was the post closed/reopened?) We'd put some work into finding a reasonable way to define "good".

For instance, "voting with the masses" seems to be a solid bet -- If you VTC on something that already has 3 or 4 votes, it's very unlikely to be "overturned" unless it's a contentious post (close/reopen/close cycle), which is pretty rare on most sites.

Again, in this case, it's far easier (and faster) to vote than to attempt to answer or edit, even when warranted.

2
  • 2
    You seem to focus pretty much solely on closing. But the reverse would have to be true as well. So re-opening would become easier too.
    – Luuklag
    Feb 13 at 20:35
  • 2
    @Luuklag While that's true, my experience is that, unless they have a strong opinion (and expertise in the particular topic beyin addressed), they tend to "trust the community" and leave it closed. This is especially the case when there have been no edits to the question in the meantime. Something that was just "wrongfully closed" tends to stay closed. The review process encourages this by making one of the reasons, "Original close reasons were not addressed". If there were no edits, then the natural conclusion is that the close reasons weren't "addressed." Feb 13 at 20:39
1

So long as it increases the diversity of reviewers, I'm onboard

Unfortunately, by the sounds of it this might achieve the opposite. Or not. It's hard to tell without more specifics. Maybe it pulls people into review if they could do more reviewing.

We have a handful of core reviewers who while appreciated, narrow down the number of people who see each review and disincentivize new reviewers starting. People are already asking Why are the review queues always empty?.

Is there any way of looking at how many people will be granted extra privileges under each of the options? I'm struggling to follow the various options.

3
  • 2
    The post says this will be a per-site setting too :) Sounds like you're on one/a few sites where opting in may not be necessary. I think for sites where even three vote closure doesn't make much of a dent in the queues, this may be an extra help. That said, yes the proposal does mean less users will see a post in the queue, which may need monitoring to see how that works out..
    – Tinkeringbell Mod
    Feb 1 at 19:41
  • @tinkeringbell or maybe if it's done on stats from outside reviews it will pull more people in? Hard to tell. Feb 1 at 19:50
  • 5
    Oh, I'm a fan of including flags/first close votes (the stuff that puts stuff in a queue in the first place) too, instead of just using the queues as a way to achieve this/use this in. You'd still end up with less users seeing a post (though bigger numbers isn't necessarily more diversity on such small scales, that's certainly less peer control).
    – Tinkeringbell Mod
    Feb 1 at 19:53
1

"What are the initial feature ideas for this?"

I have one addition, try to keep the logic simple both from a user and programming perspective. In or out of queue should be equal (much like the gold tag hammer), and I do agree (at least on Meta) that it should apply for all close reasons:

  • One gold tag badge, or
  • Two silver tag badges, or
  • One silver and two bronze tag badges, or
  • Three bronze tag badge holders can close a question; otherwise
  • 5 users, but 4 with one of them bronze and 3 with one of them silver.
4
  • 1
    This suggestion seems more complex than the recommendations above. It also seems like it would be far more powerful/impactful than what Catija recommends, which is a partial increase in voting weight for certain users.
    – TylerH
    Feb 1 at 22:55
  • 2
    I'm not sure I understand what you're saying here. Why should tag badges be involved in this at all? How do tag badges allow people to close content that's completely off topic or support smaller tags or much smaller sites? I'm hoping to find a system that is flexible enough that it works for all questions and sites equally. Tag badges don't allow for that.
    – Catija StaffMod
    Feb 2 at 5:47
  • @Catija These are ideas put forward in the first section of your question, and point 5; with much lower complexity. --- Voting weight is like flag weight of ancient times, a reinvention of the lob-sided wheel; while tag badges are directly equated with effort and expertise, thus their incorporation into the hammer.
    – Rob
    Feb 2 at 9:00
  • If you look sites with low traffic and the fewest users, for example: Tridon.SE's Top Users for tag badges, indeed there are few users with a bronze badge. Things are much worse at Freelancing.SE (a site with many questions and users), where a handful of people have bronze badges. --- No one is saying that any of these ideas cure apathy (or users who know there's a disincentive), that's a problem that needs to be addressed by a different Announcement.
    – Rob
    Feb 2 at 10:43
-1

Please make this a per-site setting

Some sites might want this, others might not. While I do realize that this slightly contradicts making it as simple a possible, there are some sites which have questions often get bounced between closed and reopened. This would only increase the problem. Also, I don't see any possible downsides from making this a per-site option, so please do so.

1
  • 8
    I've already noted in the question that it would be per-site. "Additionally, this wouldn't be shipped network-wide if we did build it. Like 3-vote close/reopen, it'd be up to the site to opt-in."
    – Catija StaffMod
    Feb 2 at 14:24
-5

When it comes to the specific criteria for part 1, counting users' close votes as multiple votes if they meet certain criteria: using the same criteria of gold tag badges (or tag badges in general) is likely not a good idea since the tag badges only count one's skill in answering questions, not asking questions.

For duplicates, basing the additional weight on answering skill makes sense - closing a question as a duplicate is basically the same as answering the question with a link, without making a statement as to whether the question is a good fit for the site or not. Users who've answered questions within the tag are more likely to know where the relevant answers to a question are, so the current hammer criteria are sensible.

On the other hand, for other close reasons, I think that the best metric to determine additional weight is one's skill in asking questions. Users who have experience in asking positively-received questions on the site are more likely to know what's a good fit for the site and what's not, since they have firsthand experience with having their questions curated by other community members. As an example, if someone's asked a question that fell just short of the guidelines, had their question closed, asked about it on the site meta, and got an explanation as to why their question isn't a good fit, the user is more likely to have an eye for future such questions.

I don't think it's a good idea to base the privilege on number of close/reopen reviews. If you are to do so, as pointed out in another answer posted while I was writing this, there are users who are heavily involved in casting close votes on questions directly ("external" votes) but don't really use the review queue. Such users would be shut out of the additional weight even though they've demonstrated casting close votes properly, which goes against the purpose of your idea.

Doing so based on "correctness" of votes is not likely a good idea, either. Often, I come across questions that users are incorrectly voting to close or reopen because of a mistaken assumption (e.g. a question that simply mentions SO that others are voting to close as site-specific, that is actually not). Even after I review to leave open, leave a comment pointing out it's on-topic, or even in some cases edit the question to make it clearer it's on-topic, the question sometimes still gets closed, probably because users follow on to the existing votes. In that case, my review would get counted as "incorrect", even though it was technically correct (the question was subsequently reopened/re-closed).

In summary, the additional weight for voting to close questions shouldn't be based on review or vote history, but instead on demonstrated knowledge of asking questions, just like how the current duplicate hammer is based on demonstrated knowledge of answering questions.

8
  • 11
    I mean... this is going to sound very egotistical/self serving, but what about those of us that physically cannot craft a useful question because every question we end up with is self solvable? I can't look at the world from a PoV that isn't mine and then generate content that would be useful to the users that make up the majority on SO. I cannot fathom how individual people can have thousands of questions asked on a site like SO. Does that make me less capable of judging whether or not a given question fits a close reason?
    – Kevin B
    Feb 1 at 18:38
  • 5
    Of course, pair that with the fact that all of my questions were bombarded over the years with downvotes due to leaving comments on posts suggesting how users can improve them.
    – Kevin B
    Feb 1 at 18:42
  • 1
    "I think that the best metric to determine additional weight is one's skill in asking questions" While this statement is true in an isolated sense, when applied in the way you seem to mean it (the score of questions the user has posted to SO) it becomes dangerously wrong. The most skilled question askers are the ones who, ideally, never have to post their questions in the first place; they are able to realize their problem or find their solution via rubber duck debugging, the Socratic method, or just thorough research. Using a negative as proof, however, is of course not at all feasible.
    – TylerH
    Feb 1 at 23:17
  • 1
    Likewise, as Kevin mentioned (and also at the risk of sounding egotistical, myself), most of my questions over the years are fairly well-formed, well-researched questions (at least once I became familiar with the site's expectations, standards, and topics). This is evidenced by the fact that--early on in the question timeline--they accrued respectable, positive scores (ranging from +1 to +8). However, as I have spent time on user moderation, I have received steady revenge downvotes on nearly all my questions, to the point where many are now at -1, -4, or even -7. (comment continued below)
    – TylerH
    Feb 1 at 23:20
  • 2
    If we are to use question score as a determining metric, even if it is weighted negatively against time so that early votes count more (which... wouldn't be fair, since it's often difficult even for employees to prove certain votes are 'revenge' votes during ad-hoc investigation, let alone via programmed metrics), that would unfairly impact users like Kevin and myself who make heavy use of user moderation tools, but pay the price of having negative question scores due to our efforts in keeping the site clean. Anyone in SOCVR can probably share their own identical story.
    – TylerH
    Feb 1 at 23:23
  • 3
    The people who are most knowledgeable about closing/reopening are the people who are engaged and informed about the specific site and its asking expectations. That's it. It doesn't matter whether they've asked, answered, or just sat around and read meta posts all day... This is why I dislike basing privileges on metrics that don't indicate any real connection to understanding the system (i.e reputation). I've seen people with thousands of rep who can't edit a post to save their life while people who are experts are stuck suggesting edits because they don't have 2k rep.
    – Catija StaffMod
    Feb 2 at 6:14
  • 3
    Most people who have a question closed don't ask about it on meta. Most people who have a question closed don't understand why the question was closed or accept that it somehow violated the rules of the site. I understand your intention here but I think your logic is making assumptions that don't have data to support it. The bulk of people who understand the rules of the site are the people who made those rules - who are usually the people answering the questions but even more often, the people working to uphold the rules.
    – Catija StaffMod
    Feb 2 at 6:16
  • 2
    Yes, sites often fracture and have factions that disagree about whether something should be closed or not - this probably isn't a great option for those sites... but for sites where the community is in agreement on these points, I think the curators - people voting to close/reopen regularly and participating in review - are the best analogues to "people who know the rules", particularly when coupled with checking their accuracy in casting those votes.
    – Catija StaffMod
    Feb 2 at 6:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .