Testing three-vote close and reopen on 13 network sites

Question

Test is live!

The 13 sites below have their votes to close set at 3. I've also reduced the votes needed for migrations to 2 (down from 3).

---

Almost two years ago I ran a test for single vote closure/reopen on Hardware Recommendations after one of their moderators reached out to me about the strain of being the person doing most of the work to close questions on that site. Shortly after that, Shog9 ran a three-vote test on Stack Overflow, which has also been deemed successful. Later that year I ran a test to do the same on Web Applications, which we later made permanent.

So, we have three sites where five votes to close aren't needed and ever since those tests I've been getting requests from various moderators to lower the number of votes on their sites. I have 26 known outstanding requests to test three-vote closure on sites and later this week - Thursday, May 6th - we'll be starting 45-day tests on half of those sites.

What's the problem?

We ended up with five votes to close based on very specific feedback from the early days of closure and the very immature state of review, though it had previously been three. Five became the default network wide and, now that we have so many more sites with different amounts of participation, it's clear that it may not be the right number any more.

What I've come to realize more recently is that we've been relying on moderators - our "exception handlers" far more than we realized to actually close questions on many of our sites - and on some sites, when the moderators aren't doing the bulk of closures, large percentages of questions that end up in the close or reopen queues never get reviewed at all, aging out entirely after a while.

There's a lot of really great thinking about this on both the MSO announcement of their initial test and the results post on MSO that I linked to above, so if you want a deeper understanding, please check out the questions and answers on those two posts - though they largely relate to the aging out issue rather than the moderators doing the bulk of the closing/reopening.

When reviews don't happen

When questions get flagged for review and those reviews age out rather being completed, that's usually because there aren't enough active, engaged users with the ability to close/reopen questions (3,000 reputation on designed sites, 500 reputation on beta and non-designed sites). On tiny sites, there simply may not be that many people using the site and on bigger sites, it's not uncommon for reviewers to just get burned out on reviewing.

Note, this is a different problem than a site where there are lots of reviews being completed but a large number of flagged posts were deemed to not close or reopen. In that case, it makes sense to understand why there's a disconnect between flaggers/voters and reviewers but the reviews are actually happening.

As an example, say that in a 60 day period 1200 questions get a flag or vote to close by someone. If 29% are closed and 3% are left open - only 32% are being handled meaning that 68% are aging out of review - we don't know whether those 68% should have been closed or not, we just know that no one had the time to act on them.

When five people are needed for this process, it puts more weight on more people to do the work and, with a relatively low number of reviews per day per person, the small crew of reviewers can become quickly overwhelmed even if they are actively reviewing posts. Reducing this number to three quickly leaves this group more room to act on more different tasks.

Assuming that all votes come from review and not on the post, lowering votes to close/reopen from five to three means a 66% increase in number of posts that can be reviewed per day with the same group of reviewers - so 15 reviewers can handle 100 questions instead of 60 with no additional work per reviewer.

When moderators do most of the closing/reopening

Many of the moderators are very reticent to unilaterally close questions - in fact, I've spoken to many who only close vote stuff that's blatantly off topic or if there are 2-3 votes from the community already, so they're expediting the process rather than dictating the scope of the site. And this is great - unfortunately, on many sites, there aren't really five active reviewers to handle many of these cases, so if questions need to be closed, it inevitably falls to the mods to do the work.

On some larger sites, mods are handling hundreds of closures per month! In some of those cases, due to low reviewing, moderators likely are acting sooner than they might otherwise. This means there's less in the review queues for users to handle so they assume there's no action to take because the moderators are doing most of the work - but there's one major flaw in this - Moderators can close questions unilaterally which can make it harder to reopen questions that have been fixed because it still takes five people to reopen that closed question - or the moderators have to do even more work by keeping up with the reopen queue, too.

Shifting sites down to three-votes to close and reopen means that it's easier for the moderators to step back and let the community handle these tasks - while posts may take longer to get closed, moderators feel less pressure to do the closing and reopening themselves and it's possible that more edited questions can get reopened.

---

Risks & Challenges

There are some things that we watch for when we're running these tests. Here's a couple of them:

• If a site does have sufficient close voters and those people are active in reviewing, this can lead to higher instances of close/reopen "wars".
• Having fewer votes to close can make it less clear which close reason is valid on posts - when you need five, it's unlikely there will be five different reasons used. When you only need three, three different reasons is more likely.

We also don't see this as a perfect solution to all things. While this has helped SO, and there's been a long-term increase in percentage of flagged posts being handled, it didn't go to 100% - many questions still age out of review. On Web Applications and Hardware Recommendations much of the closing is still done by moderators, even on Hardware Recs, where they only need one user to vote to close.

There still has to be a community willing to do the work for this to help - the site won't need five people but they will still need three or else there will still be low percentage of review completion or moderators will still need to do a lot of the work.

The Test

Starting on Thursday, May 6th for the following 45 days, we are changing the number of votes needed to close or reopen questions on the following thirteen sites to see the impact on a variety of sites. At the end of that period, we'll reset back to five and look at the data.

These sites were selected based on meta posts that were marked status-review by their moderators, indicating that they wished to participate in this test. There were 25 sites that I was aware of as of the time of writing, so this is half of the total sites that have outstanding requests for this test. I was able to group those sites in two ways - by close volume size (over a 60 day period, small <100 posts nominated for closure, medium 100<500, large 500<2000, huge >2000) and which of the two categories the site fell into.

Low review completion:

Sites where a low percentage of the posts flagged for closure were being handled at all.

Close volume size	Sites
Large	Stack Overflow en Español English Language & Usage
Medium	Server Fault Software Engineering
Small	Arduino Home Improvement

Moderator-led closure:

Sites where a high percentage of the closing/reopening was handled by moderators.

Close volume size	Sites
Huge	Stack Overflow em Português
Large	Cross Validated WordPress Development
Medium	Drupal Answers Artificial Intelligence
Small	Software Recommendations Anime & Manga

I'll be posting answers to the meta posts above on each of these sites announcing the test along with a new question on each meta just for feedback about the experience about half way in but please feel free to use this post as a way to bring broader thoughts or concerns up.

There's definitely good candidate sites that aren't on this list and we'll be moving into the next phase of this project after we look at the data from these thirteen sites, so stay tuned. If your site thinks this might help because, have a discussion on your meta - this isn't a one-time event. Once we figure out the impact of this change on sites, we'll be better able to decide when we think we can just change the setting without needing to watch things as closely.

I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.

Analysis

My plans to analyze the results are specific to the reason we're testing this.

For sites with low completion numbers, we want to see:

• increase in completed close/reopen percentages
• no increase in moderator close/reopen voting
• little or minimal increase in close/reopen warring

For sites with high moderator percentages, we want to see:

• increase in close/reopen percentages handled by community members
• -or- increase in mod votes counting as only one vote (3rd vote)
• little or minimal increase in close/reopen warring

While it'd be nice to see more reviews or more questions closed or reopened or more people participating in review, those sorts of things are secondary effects. I would hope that people feeling that their votes are effective would actually cause them to want to participate more - but I'm going to avoid hoping for that. The effect that we're really hoping for is that the same number of people will still keep doing the same number of reviews which will lead to more closures or reopenings because there's only three people needed to get there.

Thoughts?

I'm sure that lots of you have thoughts and questions for me about this - let me know and I'll work on getting answers. Are there concerns you have? Do you think this will be effective? Are we missing something important? Is there some additional analysis that you think would be beneficial?

Answer 1

What will happen on May 6th to questions with three pending close votes (or three pending reopen votes)? Will they be closed (or reopened) instantaneously? Or will they need one more vote, then the system will see they're over the threshold and close (or reopen) them?

Answer 2

OK, I'll bite.

Regarding this:

I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.

I know one of those sites (Physics). Are there more? Or is this just a polite note to us in an anonymized way? ;-). (You don't need to answer that.)

On a more serious note: what kinds of

more effective solutions to preventing answers to close-worthy questions

are you thinking of? I'm pretty sure we'd be very happy to give them a strong try.

Answer 3

I'm sure that lots of you have thoughts and questions for me about this - let me know and I'll work on getting answers.

It would be nice -if possible- to include a small digest of per site statistics in the question.

• Number of new posts vs. old posts getting flagged/vote-closed.
• Percentage of successful closures.
• Incomplete 2 vote closure.
• Percentage of reversed "leave open" items.
• Overall number of posts in the review queue per month vs completed review tasks.

We have the "Year in closing" posts, but they don't break down the correlation with items entering the queue and review tasks getting done.

Answer 4

As the volume of moderator-only closures is quite important for this topic, I would like to share an observation that Cascabel and I made a few years ago, when playing with the SE data dump for similar purposes.

I think that there should be a differentiation between moderators closing for the "duplicate" reason (and maybe also for migration) on one side, and all other closing reasons on another side. There are two reasons for the division:

• closures as a duplicate are usually clear-cut and rarely disputed.
• when the moderators are among the oldest users of the site, it is normal that they best remember having come across an older question of the same topic. This makes them naturally the ones who close the most duplicates.
• closing as a duplicate is less likely to reduce the information available on the site. If somebody sees a question they want to answer and the question is a duplicate, they can (and ideally, should) write up their answer under the duplicate target instead. This results in both more content than other closures, and less frustration among knowledgeable users.

On Cooking, we had been asking ourselves if we moderators overrule our users too much with hammers, and wanted to take a look at how much we close unilaterally. At first we were surprised at the high mod-closure rate, but once we removed the duplicate closures, we saw large changes in the numbers.

Answer 5

I'm at 70k+ on Unix & Linux but only 1500 on SuperUser. Given the high subject matter crossover I'd happily help with the review queues on SU but as yet I haven't been able to get enough points. While this is partly because I frequent one site more than the other, I'd also point a finger at the low acceptance/voting rate on SU.

Have you considered how to help review queues from others like me who (probably) have the skills to help but cannot yet do so? (Even my question/answer edits get queued for review on SU.)

Answer 6

While I can't speak for other sites, I definitely wouldn't like to see this happen on History SE.

It seems that we already have a high closure rate. One of our mods, T.E.D., posted the following in a meta answer (concerning the 3 days previous to his answer):

Percentage of new questions closed:

• History 45%
• Politics  38%
• Skeptics 25%

The very useful link provided by Shog9 in a comment below shows that History had a closure rate of 38.49% in 2020, not the highest but more than the large majority of SE sites.

Now, whether that's because History attracts more bad questions, or because History reviewers are more active, or because History reviewers are stricter in their requirements, is open to debate. I do not pretend to know the answer, but I am sure that it's not our mods who are doing the closing – they rarely close unilaterally (unless the Q is wildly off-topic or offensive). Just to be clear, I'm not anti-closing: the large majority of the closed questions deserved to be closed.

Personally, I feel that we close too many questions on History, and I know that I am not the only one who thinks this. Questions also tend to get closed very quickly, often leaving no time for the OP to edit (though it's also true that many OPs see the feedback but make no attempt to edit to avoid closure). Of course (as Mast points out in a comment below), Qs can be reopened but, as has been noted by quite a few History Meta users, that isn't easy even when an edit has fixed the main weakness.

Consequently, having a three-vote-to-close system on History would be (again, in my opinion – I do not claim to speak for everyone on our site) detrimental to History. The problems some other sites have which you mentioned in your question do not seem applicable to History.

I'm hoping that SE would not be averse to allowing some sites to maintain a 5 vote system while others have a 3 vote system.

Answer 7

Just a clarification question: there's conflicting info in your post as to when the experiments will start on all sites.

At the top of the post, you say:

I have 25 known outstanding requests to test three-vote closure on sites and later this week - Thursday, May 6th - we'll be starting 45-day tests on half [12] of those sites.

But later on, you say:

Starting on Monday, May 3rd for the following 45 days, we are changing the number of votes needed to close or reopen questions on the following twelve sites to see the impact on a variety of sites.

Will the experiment begin today or on Thursday?

Answer 8

I don't think that a question should be closed or opened by three "random" votes. On the other hand, we could "get there" in a roundabout way by giving certain individuals "double" votes.

This would be a user privilege, whereby users with 10,000 rep (or some other threshold) could effectively cast TWO close or reopen votes instead of one. THREE such users would cast six votes. Or it could be two double voters and one single voter, or one double voter and three single voters or the current five single voters to add up to a total of five.

That way, we would retain the "fabric" of five close or re open votes, while allowing close or reopening by three "senior" users instead of five users or one moderator.

Another possibility is to allow moderators to choose to cast a "double" vote instead of "five" votes. Many moderators shy away from unilaterally taking "five" vote actions in order to defer to the community, but would welcome the chance to "personally" cast single votes, and probably would welcome a chance to cast a double (multiple) vote that recognizes their seniority. Of course, they would retain the right to cast "unilateral" votes in the case of spam or bad (not just mediocre) content.

Edit: New sites in their "start up" period would have reduced thresholds for "double votes" on closing and reopening, just as "single" votes now do.

Answer 9

Thanks for this thorough and very well-written post! I'm not complaining about what is being done on those 13 sites, but I would very much not like this to change on some of the newer Beta sites like Matter Modeling, because we're still getting a feel for what the community wants closed and not closed, and some posts have more than 3 people with an opinion on either side.

Furthermore, people can cast close/re-open votes with only 500 reputation on such Beta sites, and it's extremely easy to reach 500 for users that participate in the Private Beta because voting is rampant in the first 2 weeks (just look at the voting analytics of Beta sites from the first 2 weeks compared to later on). Some of those Private Beta users who reached 500 rep with only a couple posts that were upvoted 25 times each, may have too big of an influence if they can cast 1/3 of the required close/re-open votes, even after years of not participating in the site at all since Private Beta.

Answer 10

This seems to boil down to "high-rep users are not toiling hard enough in the moderation queues." To me this means incentives are misaligned. I know that badges are the current carrot to induce users to do this work, but I also know that badges aren't enough to get me to review very many posts.

Why would a high-rep user want to deal with a stream of generally awful posts every day? What would make it more bearable? What would make it fun? In the past you could at least have the satisfaction of informing a homework poster that they are a moron, but we are told to be nice to new users now. :-) But after composing a thoughtful and patient answer to a question only to see it read and immediately deleted so that the homework poster can get away with cheating, a high-rep user is in no mood to deal with more of the same in the moderation queues.

So. Figure out a way to make cleaning the Augean stables stimulating and you won't have any problem finding five votes to close or reopen posts. My wife's foreign language course uses literal bells and whistles to encourage her to keep plugging away at her lessons. I don't know if that's the right carrot for this site, but it is clear that the virtual pay is too low.

Answer 11

I'm going to interpret this as a formal review request that has been actually actioned on my A&M Meta question.

I'm glad to see that I received an answer, and I'm happy also to see that this is being expanded in the relatively sensible fashion that has been laid out.

I'm bitter behind the fact that I had to wait over a year to hear anything about this, to the degree that I'm concerned that the efficacy of this test will falter due to lack of participants, which can only undermine both the value of the feature and the data in the test.

Answer 12

It might be very interesting to compare SE with its antithesis, USENET. It's hard to earn a "forged cancel" there at all (you must be a hardboiled spammer), and in theory even crackpots flooding a group can be silenced by killfiling, i.e. the work needed is on the casual user, not a mod.

Thus, fundamentals shall be considered first: How many low quality remaining on a SE will hurt it in the long run (in contrast to, say, them simply being ignored)? With a high inflow of good questions, they evidently seem less a problem to me. (For a few random examples of different categories I know, sci.math died the crackpot death, the German SF group is clean but lacks interest, the German politics groups are a cesspool, but the German joke group kinda survives up to now, conflicting interests declared.)

I clearly can relate to SE officials seeing the downfall of USENET and prefer to err on the side of caution in return. More conflicting interests declared: If I enter a group and get my first post nuked for being unclear, it's rather improbable that I try to reword it into clearness and blame the thematic itself to be unclear. (Questions are brain-children, insult them and you insult the parent :-)

To sum up my personal stance on the matter: If in Rome etc. - I'm just a customer here.

Answer 13

After the 45 day test period, the system will return to [at least until policy is permanently changed] requiring 5 votes being needed to close/reopen.

Question: will posts closed with only 3 votes during the 45 day test window reopen after 45 days when the threshold returns to 5 votes? Or will they remain closed?

Answer 14

How can a site be nominated for this feature? I asked at our meta if people agree with the request. Should moderators ask it to the staff?

By the way, at Earth Science what we find difficult is to delete questions, much more than closing them. Is it possible to reduce the number of deleting votes as well?

Answer 15

Will site moderators have the option of choosing for their own site 3 vs 5?

I think this is a great idea, but I would like to ask if the feature will be hardwired, or if moderators will have the ability to choose 3 vs 5 votes? In an ideal world, participation is encouraged and increased such that going back to 5 votes makes sense. After all, we're not just trying to block poor questions before they get too involved, we're also trying to train our stack users to become involved in the culture of the site, both by processing reviews and by participating in the meta discussions that lead to cultural refinement.

If this proves successful, do we hardwire all of Stack Exchange to 3 votes, regardless of any stack's ability to educate and motivate their user bases? Or will the mods have the ability to switch back and forth as they see fit to accommodate the amount of support they're receiving?

Answer 16

Are there Stack Exchange sites where 3 votes are sufficient for closure but 5 are required for reopening? I seem to have observed this on Chess.SE. What's the reason for this asymmetry?

Answer 17

We really need a better system.

So I'm seeing two problems, overall, with closing, and they are actually stemming from the same issue: 1) "Close queues are huge and no one wants to look at them anymore" and 2) typically on smaller sites, "too many questions are being closed for dubious reasons because you need so few people to vote for closure".

As I see it, they both have the same core problem, which is that close votes can easily just be a system for trolling, and seem to have become so on some sites.

You'd like to think that at 3k rep, you'd have responsible, interested users who want to keep their community inviting and on-topic but I think what we're seeing is that this isn't really true, or at any rate, that "I'm helping!" can very easily mean creating a bigger problem than you're solving by being overly happy with that VTC button. You've just put something in the queue and potentially created a bigger problem than the question would ever have been.

It would be interesting to know what percentage of posts on Stack Overflow get a single VTC. Because that's what's clogging up the queue. You can end up with a question that actually, really, definitely need to be closed, but it's buried behind what a thousand people thought maybe was kind of marginally against some rule that exists in their head, so they clogged up the system with their VTC.

I don't really know what the solution is, but I think Hauke's answer was tickling the right spot. Serious problem questions already get tackled with gusto by the rank and file, who will downvote. Or boring, unfocused, unanswerable questions tend to get skipped over with no votes. It seems like we should be able to leverage this existing system in some way.

Just to brainstorm a little, what if it worked more like....

• You VTC a question. It is now on the queue.
• The question is downvoted by a rank and file user. This counts as an additional close vote. So a VTC + 4 downvotes will close the question.
• Similarly, if the question is upvoted by a rank and file user (or if an answer is posted), then the VTC is removed from the queue. The VTC still stands but it's no longer clogging up the queue with questions that most people actually liked and upvoted. (And since it's no longer on the queue, downvotes will no longer count as close votes. Someone else must VTC to get that ball rolling again. Basically, errant one-off VTCs are easily gotten rid of.)

Votes to reopen would work similarly. 3k rep is required to initiate but then upvotes by any user will count as a re-open vote, making bad close decisions easier to undo by the community as a whole. Downvotes on a request to reopen remove it from the request to reopen queue.

Same mechanics, but far easier to deal with, and "rank and file" community input counts a lot more and helps automate the system.

Answer 18

Personally, I would like to see the the vote count be higher on sites like Stack Overflow. I have had (and have seen) many questions immediately shot down due to easy to fix things like missing information or a lack of understanding. I feel like this is a waste of potential, as (in my eyes at least), many of those questions could have had a good outcome had people not been so quick to close (or, in this case, had been harder to close such as needing more votes). Just my two pennies' worth.