188

Test is live!

The 13 sites below have their votes to close set at 3. I've also reduced the votes needed for migrations to 2 (down from 3).


Almost two years ago I ran a test for single vote closure/reopen on Hardware Recommendations after one of their moderators reached out to me about the strain of being the person doing most of the work to close questions on that site. Shortly after that, Shog9 ran a three-vote test on Stack Overflow, which has also been deemed successful. Later that year I ran a test to do the same on Web Applications, which we later made permanent.

So, we have three sites where five votes to close aren't needed and ever since those tests I've been getting requests from various moderators to lower the number of votes on their sites. I have 26 known outstanding requests to test three-vote closure on sites and later this week - Thursday, May 6th - we'll be starting 45-day tests on half of those sites.

What's the problem?

We ended up with five votes to close based on very specific feedback from the early days of closure and the very immature state of review, though it had previously been three. Five became the default network wide and, now that we have so many more sites with different amounts of participation, it's clear that it may not be the right number any more.

What I've come to realize more recently is that we've been relying on moderators - our "exception handlers" far more than we realized to actually close questions on many of our sites - and on some sites, when the moderators aren't doing the bulk of closures, large percentages of questions that end up in the close or reopen queues never get reviewed at all, aging out entirely after a while.

There's a lot of really great thinking about this on both the MSO announcement of their initial test and the results post on MSO that I linked to above, so if you want a deeper understanding, please check out the questions and answers on those two posts - though they largely relate to the aging out issue rather than the moderators doing the bulk of the closing/reopening.

When reviews don't happen

When questions get flagged for review and those reviews age out rather being completed, that's usually because there aren't enough active, engaged users with the ability to close/reopen questions (3,000 reputation on designed sites, 500 reputation on beta and non-designed sites). On tiny sites, there simply may not be that many people using the site and on bigger sites, it's not uncommon for reviewers to just get burned out on reviewing.

Note, this is a different problem than a site where there are lots of reviews being completed but a large number of flagged posts were deemed to not close or reopen. In that case, it makes sense to understand why there's a disconnect between flaggers/voters and reviewers but the reviews are actually happening.

As an example, say that in a 60 day period 1200 questions get a flag or vote to close by someone. If 29% are closed and 3% are left open - only 32% are being handled meaning that 68% are aging out of review - we don't know whether those 68% should have been closed or not, we just know that no one had the time to act on them.

When five people are needed for this process, it puts more weight on more people to do the work and, with a relatively low number of reviews per day per person, the small crew of reviewers can become quickly overwhelmed even if they are actively reviewing posts. Reducing this number to three quickly leaves this group more room to act on more different tasks.

Assuming that all votes come from review and not on the post, lowering votes to close/reopen from five to three means a 66% increase in number of posts that can be reviewed per day with the same group of reviewers - so 15 reviewers can handle 100 questions instead of 60 with no additional work per reviewer.

When moderators do most of the closing/reopening

Many of the moderators are very reticent to unilaterally close questions - in fact, I've spoken to many who only close vote stuff that's blatantly off topic or if there are 2-3 votes from the community already, so they're expediting the process rather than dictating the scope of the site. And this is great - unfortunately, on many sites, there aren't really five active reviewers to handle many of these cases, so if questions need to be closed, it inevitably falls to the mods to do the work.

On some larger sites, mods are handling hundreds of closures per month! In some of those cases, due to low reviewing, moderators likely are acting sooner than they might otherwise. This means there's less in the review queues for users to handle so they assume there's no action to take because the moderators are doing most of the work - but there's one major flaw in this - Moderators can close questions unilaterally which can make it harder to reopen questions that have been fixed because it still takes five people to reopen that closed question - or the moderators have to do even more work by keeping up with the reopen queue, too.

Shifting sites down to three-votes to close and reopen means that it's easier for the moderators to step back and let the community handle these tasks - while posts may take longer to get closed, moderators feel less pressure to do the closing and reopening themselves and it's possible that more edited questions can get reopened.


Risks & Challenges

There are some things that we watch for when we're running these tests. Here's a couple of them:

  • If a site does have sufficient close voters and those people are active in reviewing, this can lead to higher instances of close/reopen "wars".
  • Having fewer votes to close can make it less clear which close reason is valid on posts - when you need five, it's unlikely there will be five different reasons used. When you only need three, three different reasons is more likely.

We also don't see this as a perfect solution to all things. While this has helped SO, and there's been a long-term increase in percentage of flagged posts being handled, it didn't go to 100% - many questions still age out of review. On Web Applications and Hardware Recommendations much of the closing is still done by moderators, even on Hardware Recs, where they only need one user to vote to close.

There still has to be a community willing to do the work for this to help - the site won't need five people but they will still need three or else there will still be low percentage of review completion or moderators will still need to do a lot of the work.

The Test

Starting on Thursday, May 6th for the following 45 days, we are changing the number of votes needed to close or reopen questions on the following thirteen sites to see the impact on a variety of sites. At the end of that period, we'll reset back to five and look at the data.

These sites were selected based on meta posts that were marked by their moderators, indicating that they wished to participate in this test. There were 25 sites that I was aware of as of the time of writing, so this is half of the total sites that have outstanding requests for this test. I was able to group those sites in two ways - by close volume size (over a 60 day period, small <100 posts nominated for closure, medium 100<500, large 500<2000, huge >2000) and which of the two categories the site fell into.

Low review completion:

Sites where a low percentage of the posts flagged for closure were being handled at all.

Moderator-led closure:

Sites where a high percentage of the closing/reopening was handled by moderators.

I'll be posting answers to the meta posts above on each of these sites announcing the test along with a new question on each meta just for feedback about the experience about half way in but please feel free to use this post as a way to bring broader thoughts or concerns up.

There's definitely good candidate sites that aren't on this list and we'll be moving into the next phase of this project after we look at the data from these thirteen sites, so stay tuned. If your site thinks this might help because, have a discussion on your meta - this isn't a one-time event. Once we figure out the impact of this change on sites, we'll be better able to decide when we think we can just change the setting without needing to watch things as closely.

I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.

Analysis

My plans to analyze the results are specific to the reason we're testing this.

For sites with low completion numbers, we want to see:

  • increase in completed close/reopen percentages
  • no increase in moderator close/reopen voting
  • little or minimal increase in close/reopen warring

For sites with high moderator percentages, we want to see:

  • increase in close/reopen percentages handled by community members
  • -or- increase in mod votes counting as only one vote (3rd vote)
  • little or minimal increase in close/reopen warring

While it'd be nice to see more reviews or more questions closed or reopened or more people participating in review, those sorts of things are secondary effects. I would hope that people feeling that their votes are effective would actually cause them to want to participate more - but I'm going to avoid hoping for that. The effect that we're really hoping for is that the same number of people will still keep doing the same number of reviews which will lead to more closures or reopenings because there's only three people needed to get there.

Thoughts?

I'm sure that lots of you have thoughts and questions for me about this - let me know and I'll work on getting answers. Are there concerns you have? Do you think this will be effective? Are we missing something important? Is there some additional analysis that you think would be beneficial?

28
  • 15
    In SOpt, we discussed that about a year ago (with relative community support on lowering the threshold). One of our mods even questioned one CM about that, and we've never got any response. And now I'm surprised that you're not only doing this test, but also ignored SOpt. You (the CM team) could've said "we're working on it" in our meta, at least we'd know our request wasn't ignored, but instead you left the impression that you simply ignored us. I'm really disappointed. – hkotsubo May 4 at 12:47
  • 8
    @hkotsubo I'm sorry - but, while I did get a note from JNat back in January 2020, the last thing I heard was that y'all were still discussing it and that no decision had been made - so there wasn't anything for us to respond about and I didn't hear anything else that I can remember. Since that time, we rolled out a way for mods to draw attention to things that need our help - using the status-review tag. When I went to make a list of sites to consider for this test, I missed SOpt because that tag wasn't used. – Catija May 4 at 13:42
  • 11
    And... not that it's a great excuse because I should have said something - I haven't responded to any of these posts as far as I can remember. I'm very frustrated that it's taken so long to get this project moving and I apologise for that but I had no way of knowing when it would actually happen. I did write an answer here on MSE when someone asked what was going on and I have tried to be communicative with the mods who have asked me about it directly. We've been incredibly short-handed the last year and it's only now, with the new hires, that we're finally able to get things moving. – Catija May 4 at 13:47
  • 3
    @hkotsubo Why would it not be possible if we've done it on SO and other sites? We can't make it asymmetric, with close requiring a different number of votes as reopen, but there's nothing preventing y'all from getting this changed other than us needing to find the time to actually test it. I looked at the numbers and it does look like the mods are doing a ton of work and so I am going to get this going on SOpt as part of the test - with the big caveat that I'm concerned that it won't show much improvement if the users don't actually do any of the work. – Catija May 4 at 14:27
  • 5
    @Catija some years ago, Cascabel and me tried calculating our own mod closing percentage on cooking, to see if we are very likely to overrule the community :) and noticed quickly that it makes a lot of sense to distinguish between a mod hammer for duplicates and a mod hammer for one of the other reasons. Dupes are usually more clear-cut, and also require the memory of "I know I have seen this", which our current high rep users don't have, since they have been around for a shorter time than the mods. Did you make that distinction in counting the mod-close percentage? – rumtscho May 4 at 15:39
  • 1
    @rumtscho That's a good thing to think about - all of the close reasons are lumped together and I haven't broken them out that way but it's a good idea to do that with duplicates (and maybe migrations?) But I think those are the only ones that may need special consideration. In all, over 60 days, the Cooking mods have closed 46 and the community 20. :) – Catija May 4 at 15:45
  • 6
    Glad to see we (SOpt) will participate. Thanks @hkotsubo and Catija. – Bacco May 4 at 17:17
  • 2
    @anonymous2 Yeah, I'll be trying to be certain we only consider mod hammers - so fifth or third votes would be excluded from the count. I know a lot of mods are OK with those votes but, on the scales I'm seeing, the votes tend to be unilateral or second votes at best in many cases. :) – Catija May 5 at 17:08
  • 2
    @hkotsubo The data from the first few days of any test like this can be somewhat misleading - the test itself can cause additional awareness and participation that doesn't reflect the natural behaviors of people on the site. As such the plan is to not actually look at the first week or so of data when assessing the outcomes. The 45-days includes a buffer so that I can grab a nice, clean 30-day period in the middle or so. Regardless, the site settings have been changed at this point. – Catija May 6 at 17:35
  • 3
    Why have the migration votes been changed and why didn’t the sites that are the most frequent migration targets get to weigh in on that decision? ELL has had trouble in the past with very low quality migrations from ELU, so the ELL community should have had some input on whether that was a good idea. – ColleenV May 6 at 19:26
  • 3
    @Catija Is there any indication that the current migration votes need adjusting? So what if some questions get closed because they’re low quality instead of migrated? You’re assuming the amount of questions currently getting migrated needs to be adjusted upwards, when honestly there’s still quite a few stinkers getting through, and dropping the requirement is going to make it harder for people to stop the migration of crap. – ColleenV May 6 at 22:01
  • 4
    It also occurs to me that while it’s easier than ever to migrate stuff to ELL, we still need the same number of votes to reject a migration. Changes to migration paths should involve both sites, even if it’s just a “hey, this is changing” post. – ColleenV May 6 at 22:38
  • 2
    I'm having to balance things here. It's much more similar to have the 2/3 ratio vs 3/5 ratio rather than having 3/3. One person could prevent all migrations simply by using any other close reason - whether intentionally or accidentally. You're talking about it like I've made it impossible to prevent migrations rather than considering the other possibility. Honestly, I've really never liked migrations between ELL and ELU because they tend to be terrible quality and because there's not enough participation on ELL to fight them, @ColleenV - much rather just go back to mod-only migrations. – Catija May 6 at 22:42
  • 5
    @Catija I’m not really asking for any particular action here. I’m giving feedback about how the decision was handled. I don’t know whether it will be a burden or not. Of the 11 that were split decisions, how many were well-received on ELL? I don’t have the reputation to see the migrated away page on ELU, but this query I cobbled together a couple years ago shows that 1/3 or less of the migrations get an upvote on ELL. Counting votes on ELU doesn’t measure “good” migrations. – ColleenV May 7 at 11:08
  • 3
    Now I’ve read multiple times that moderators (and also gold badgers) aren’t always comfortable with using their unilateral closing powers all the time. Is it time to reconsider these proposals: How Do I Opt Out of Privileges? and Add a way for moderators to cast a normal, non binding close/open vote? Sure, the binding close-vote is a privilege, but it is also a burden — an unnecessary one. Just give these privileged users a choice: binding vote or regular vote. I don’t see any problems in this, only solutions to existing problems. – Sebastian Simon May 11 at 19:51

17 Answers 17

48

What will happen on May 6th to questions with three pending close votes (or three pending reopen votes)? Will they be closed (or reopened) instantaneously? Or will they need one more vote, then the system will see they're over the threshold and close (or reopen) them?

4
  • 19
    Hopefully the check is closeVotes >= 3 and not closeVotes == 3, or else infinitely many close votes may accumulate on a question! But in seriousness, I believe the threshold is checked at each vote and the question becomes closed once the amount of close votes is over the threshold. – Sebastian Simon May 4 at 11:40
  • 15
    I'm getting it confirmed but we only check the status when new votes happen, so the most likely outcome is that the next vote to close will satisfy the requirements and cause the post to be closed. That means that some posts with 3 already may end up aging out if they don't get those final votes. Hopefully that won't happen much? – Catija May 4 at 15:43
  • 8
    I've just confirmed that the check is votes >= 3. There was a question with 3 close votes (before config was changed), I voted and it was closed. I believe it's a corner case, but I'm curious to know how/if those cases of questions closed with 4 votes will interfere in the analysis – hkotsubo May 6 at 18:49
  • 6
    @hkotsubo Given Catija’s comment — “the plan is to not actually look at the first week or so of data when assessing the outcomes. The 45-days includes a buffer so that I can grab a nice, clean 30-day period in the middle or so.” —, the effect on the analysis will probably be very minimal. – Sebastian Simon May 6 at 18:59
25

OK, I'll bite.

Regarding this:

I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.

I know one of those sites (Physics). Are there more? Or is this just a polite note to us in an anonymized way? ;-). (You don't need to answer that.)

On a more serious note: what kinds of

more effective solutions to preventing answers to close-worthy questions

are you thinking of? I'm pretty sure we'd be very happy to give them a strong try.

11
  • 8
    Physics is definitely not the only site - some of the requests that I have read specifically mention wanting to close posts more quickly before being answered. That said, I did specifically think of Physics when writing that paragraph. I've been struggling with this for a long time because I know y'all really want this and I understand that but - if my queries are correct - the number of questions answered prior to a third vote to close is still really high, so I'm not sure that this will do what y'all are hoping - it might help but, yeah, I think there may be more direct solutions. – Catija May 4 at 16:06
  • 3
    Of course, we don't want anyone getting piled on for answering a question that they didn't realize was out of scope or wasn't sufficiently detailed. Or, on many sites, there's disagreement about what sorts of questions should be answered - for example, questions that just want an answer without showing any attempt to self-solve. These are often considered "homework questions" - the sort of question that looks like it was copied right out of an assignment and the user doesn't show their attempts to figure it out on their own. And, there may not be a great solution to that, sadly. – Catija May 4 at 16:08
  • I think that - and y'all probably have analyzed it a lot - understanding what questions are getting answered before being closed and looking for commonalities - types of question, person answering (or type of person such as a new user, experienced user, etc), reason for closure... etc. To understand where the breakdown in communication is. Maybe you need better guidance to prevent the questions from being asked such as a customized Ask modal or sidebar (view the ask page in incognito mode to see the modal). Maybe better education for the people answering about when not to answer... – Catija May 4 at 16:11
  • 1
    OK, I lied, y'all require registration to ask, so you can't just view in incognito, so here's some screenshots of the default (what y'all have) and what it can look like when customized (Code Review). It's possible a more descriptive modal may guide people on how to ask better. The right nav bar can also be customized with helpful links and instructions. While we can't guarantee people will read them, we can at least make it clear what they need to do. – Catija May 4 at 16:15
  • 3
    @Catija We should indeed think very carefully about what we put into that modal, and how we can use that to improve new users' questions. But in terms of questions getting answers before they can get closed, it feels to me that if it's an education (or communication) problem, then the education should be focused on the users writing answers, not the askers. – E.P. May 4 at 16:20
  • 1
    So, say, something like a modal, or ideally a warning box that appears above the answer text editor box (even better, something that pops in once the user clicks on the editor box), which reminds the user what kinds of h/w questions are on-topic and which ones shouldn't be answered, and which appears, say, on questions with the h/w tag or which have one or more pending close votes with that close reason. How feasible would something like that be? – E.P. May 4 at 16:22
  • 2
    That's an option - and, honestly, both is good here! But it's important to understand the answerers' motivations. Do they disagree with the questions being close-worthy? Are they unaware that the question should be closed? Do they have thoughts on how they can improve the question - it's possible that the best case scenario for the asker and the answerers is that the questions get improved first and answered - so it's not so much that they can't answer the question but that they should work with the asker to make it a good question before offering a (possibly problematic) answer. – Catija May 4 at 16:24
  • 1
    ^^^ That was written before your prior comment so assume it's a reply to the other one. – Catija May 4 at 16:24
  • 1
    More generally, when you say 'better education for the people answering about when not to answer', that feels like the most critical step to me. But it's not an easy population to reach (i.e., to reach in an effective and targeted way, and at a time when they're receptive to the message and the message is actionable). (← also written before your last comment =).) – E.P. May 4 at 16:25
  • Regarding your last comment -- yeah, I'm not sure that we have a grasp on how that population really looks like in detail that's fully grounded in evidence. We should probably use the time while the current experiment runs to dig into that and understand our problem better. – E.P. May 4 at 16:27
  • 1
    I think this will be a very significant improvement for sites like raspberrypi.stackexchange.com where +/- voting is very stingy, resulting in there being far too few people who can access the close/reopen queues. This was completely dsyfunctional for years, thank you, but I am honestly appalled that it took you this long to bother noticing. My impression for a long time has been that management does not care at all about the smaller sites. Hopefully the beginning of something new there. – goldilocks May 8 at 16:06
18

I'm sure that lots of you have thoughts and questions for me about this - let me know and I'll work on getting answers.

It would be nice -if possible- to include a small digest of per site statistics in the question.

  • Number of new posts vs. old posts getting flagged/vote-closed.
  • Percentage of successful closures.
  • Incomplete 2 vote closure.
  • Percentage of reversed "leave open" items.
  • Overall number of posts in the review queue per month vs completed review tasks.

We have the "Year in closing" posts, but they don't break down the correlation with items entering the queue and review tasks getting done.

1
  • 4
    I'll see what I can do - a lot of these don't align specifically with the queries I'm using, so I'd have to get someone to write the queries for me since I'm a SQL novice. – Catija May 5 at 17:10
17

As the volume of moderator-only closures is quite important for this topic, I would like to share an observation that Cascabel and I made a few years ago, when playing with the SE data dump for similar purposes.

I think that there should be a differentiation between moderators closing for the "duplicate" reason (and maybe also for migration) on one side, and all other closing reasons on another side. There are two reasons for the division:

  • closures as a duplicate are usually clear-cut and rarely disputed.
  • when the moderators are among the oldest users of the site, it is normal that they best remember having come across an older question of the same topic. This makes them naturally the ones who close the most duplicates.
  • closing as a duplicate is less likely to reduce the information available on the site. If somebody sees a question they want to answer and the question is a duplicate, they can (and ideally, should) write up their answer under the duplicate target instead. This results in both more content than other closures, and less frustration among knowledgeable users.

On Cooking, we had been asking ourselves if we moderators overrule our users too much with hammers, and wanted to take a look at how much we close unilaterally. At first we were surprised at the high mod-closure rate, but once we removed the duplicate closures, we saw large changes in the numbers.

14

I'm at 70k+ on Unix & Linux but only 1500 on SuperUser. Given the high subject matter crossover I'd happily help with the review queues on SU but as yet I haven't been able to get enough points. While this is partly because I frequent one site more than the other, I'd also point a finger at the low acceptance/voting rate on SU.

Have you considered how to help review queues from others like me who (probably) have the skills to help but cannot yet do so? (Even my question/answer edits get queued for review on SU.)

9
  • 14
    There's been some (though not a ton) of conversation internally about whether reputation is the best signal that someone should have certain privileges. E.g. Someone can have 1k rep and still be fully capable of editing posts without review but someone with 10k rep may not have ever edited a single post - so may not know how to do it well. I would put your question in the same bucket. We want to ensure that reviewers understand how to review in general but they also need to be aware of the norms/expectations of the specific site (what is close-worthy and what isn't). – Catija May 5 at 15:15
  • 7
    So, we've generally resisted saying "you have 3k rep on Site A, so that should be enough, so you can review everywhere" - because of that second part - the site norms/expectations... but, again... someone doesn't need 3k to know those and having 3k doesn't absolutely mean that you're aware of them. The question then becomes... so... what do we use instead that's not a pain in the butt to validate. And... that's hard. For edits it's easy - say, you've had 90% of your suggested edits approved and you've submitted 50, so we trust you now! – Catija May 5 at 15:18
  • 2
    I'm not sure it's so easy for review, though... review is on/off - there's flags for some queues (LQP, Close/reopen) but there are extenuating circumstances that could make flags disputed (such as the post being edited in the interim)... we could do some sort of training that, people who completed it would be allowed - but that's new and big and complex (I'd assume) so might be really slow to build, though it might make review, overall better - we'd also have to make it highly customizable per-site, which makes it even harder, since we're talking about 170 sites! – Catija May 5 at 15:19
  • 7
    As a note, I'm (personally) less concerned about low accept rate - I'm not a big proponent of thinking too hard about accepted answers because the asker isn't always the best judge of the value of an answer, though that can depend on the site. What I am concerned about is the low voting - experts on the site should be there to review/validate the posts on the sites and should be voting to indicate quality and correctness (or lack thereof). – Catija May 5 at 15:31
  • @Catija, if I change the phrase "low acceptance rate" to "low voting/acceptance rate" it even more strongly underscores my point, I think – roaima May 5 at 17:14
  • 1
    @Catija for review training maybe you can give <3K people 100 review audits, and if they pass most of them they get the privilege? The audit system is already in place on SO and could presumably be implemented on other sites without needing to build something "new and big and complex". – Marijn May 5 at 20:24
  • @Marijn my edits are taking ~ 2 days to get through the review queue on SU. Surely that shouldn't be normal? – roaima May 5 at 20:46
  • 2
    @roaima Considering the number of people complaining that edits are blocked entirely because the queue is full, it's... something? :D As someone who got my first rep on SO through edits, I understand - even three years ago it was taking 8 hours up to a couple of days to get through review. The one thing about suggested edits, though, is that at least they don't expire, so the major blocker is just that - if the queue gets too full and blocks suggestions. (Ah, I see now you were talking about SU, not SO - but I think there's a similar cause - not a ton of reviewers). – Catija May 5 at 23:14
  • 2
    @Marijn You'd need to overhaul the audit system first. A significant number of audits are bad. – TylerH May 13 at 14:21
13

Just a clarification question: there's conflicting info in your post as to when the experiments will start on all sites.

At the top of the post, you say:

I have 25 known outstanding requests to test three-vote closure on sites and later this week - Thursday, May 6th - we'll be starting 45-day tests on half [12] of those sites.

But later on, you say:

Starting on Monday, May 3rd for the following 45 days, we are changing the number of votes needed to close or reopen questions on the following twelve sites to see the impact on a variety of sites.

Will the experiment begin today or on Thursday?

2
  • 25
    Fixed - dates slipped and I only changed the info in one place rather than two. – Catija May 3 at 20:18
  • 6
    if the fix is applied, should we downvote this answer? – Ooker May 9 at 18:35
13

While I can't speak for other sites, I definitely wouldn't like to see this happen on History SE.

It seems that we already have a high closure rate. One of our mods, T.E.D., posted the following in a meta answer (concerning the 3 days previous to his answer):

Percentage of new questions closed:

  • History 45%
  • Politics  38%
  • Skeptics 25%

The very useful link provided by Shog9 in a comment below shows that History had a closure rate of 38.49% in 2020, not the highest but more than the large majority of SE sites.

Now, whether that's because History attracts more bad questions, or because History reviewers are more active, or because History reviewers are stricter in their requirements, is open to debate. I do not pretend to know the answer, but I am sure that it's not our mods who are doing the closing – they rarely close unilaterally (unless the Q is wildly off-topic or offensive). Just to be clear, I'm not anti-closing: the large majority of the closed questions deserved to be closed.

Personally, I feel that we close too many questions on History, and I know that I am not the only one who thinks this. Questions also tend to get closed very quickly, often leaving no time for the OP to edit (though it's also true that many OPs see the feedback but make no attempt to edit to avoid closure). Of course (as Mast points out in a comment below), Qs can be reopened but, as has been noted by quite a few History Meta users, that isn't easy even when an edit has fixed the main weakness.

Consequently, having a three-vote-to-close system on History would be (again, in my opinion – I do not claim to speak for everyone on our site) detrimental to History. The problems some other sites have which you mentioned in your question do not seem applicable to History.

I'm hoping that SE would not be averse to allowing some sites to maintain a 5 vote system while others have a 3 vote system.

11
  • 19
    We have no plans to change this on sites that don't wish us to. – Catija May 4 at 2:31
  • 9
    FWIW, you might find this interesting: meta.stackexchange.com/questions/359735/2020-a-year-in-closing (History is 38% for questions asked in 2020). That said, this isn't really about getting more questions closed; that may be an outcome on some sites, but at the scale of a site like History I doubt there are so many questions being flagged or voted on that moderators and trusted users can't already close all that they would like to (and thus, reducing the threshold would have little or no effect on volume). – Shog9 May 4 at 3:26
  • 16
    Closing a question isn't the end of the world. Many questions can be improved enough to warrant reopening, but this takes some effort (either from OP or the community). Edits can still come in after closure, successful edits even automatically put the question into the reopen queue. – Mast May 4 at 3:50
  • @Catija Good to hear there's flexibility there. I suspect most History SE folk are happy with things as they are (but I could be wrong). – Lars Bosteen May 4 at 7:00
  • 1
    @Mast True, but on History (at least, and as has been noted by quite a few users on our site) it's difficult to get questions reopened. – Lars Bosteen May 4 at 7:07
  • 4
    @LarsBosteen maybe less so if it only takes 3 people to reopen? – Martin Smith May 4 at 11:56
  • 5
    @Mast Only the first edit after closure puts it into the reopen queue. It's a big and well-known problem in the system that more effectively prevents OP edits from getting questions reopened than anything else. – TylerH May 4 at 16:08
  • 1
    Definitely true for SE History. It's a bit disheartening to see how many questions get closed, especially on new users. Some certainly deserve it, some closures however are much more debatable and lowering the limit will make it worse. – Italian Philosophers 4 Monica May 6 at 18:19
  • 3
    One thing I've noted on History SE is that there are five (or so) very active non-moderator members that VTC questions are a "group," and questions that go against their (collective) taste tend to get closed. – Tom Au May 9 at 4:37
  • @TomAu True. It would be nice to see more members using their voting rights to make the process more representative of the site in general. – Lars Bosteen May 9 at 5:04
  • 1
    The high closure rate on history is a good thing. A lot lf low quality input arrives, and it needs to be removed, or, the questions improved and then reopened. The high noise to signal ratio is why I don't visit as often as I used to. – KorvinStarmast May 12 at 12:05
7

Thanks for this thorough and very well-written post! I'm not complaining about what is being done on those 13 sites, but I would very much not like this to change on some of the newer Beta sites like Matter Modeling, because we're still getting a feel for what the community wants closed and not closed, and some posts have more than 3 people with an opinion on either side.

Furthermore, people can cast close/re-open votes with only 500 reputation on such Beta sites, and it's extremely easy to reach 500 for users that participate in the Private Beta because voting is rampant in the first 2 weeks (just look at the voting analytics of Beta sites from the first 2 weeks compared to later on). Some of those Private Beta users who reached 500 rep with only a couple posts that were upvoted 25 times each, may have too big of an influence if they can cast 1/3 of the required close/re-open votes, even after years of not participating in the site at all since Private Beta.

3
  • 5
    Each site will be able to request it on a one-off basis. As I mentioned on Lars' answer, we're not going to force any site into this. But I'm also not going to blanket promise not to make this change on newer sites if they ask for it. – Catija May 10 at 14:11
  • Regarding "some posts have more than 3 people with an opinion on either side", questions can be reopened once they're closed, and closed again; it's natural if the community agrees or disagrees on closure... it happens all the time on bigger sites (especially if said posts get edited/improved during closure). Users can't vote to close questions they've already successfully closed, so everyone with the rep requirement gets an equal opportunity to say whether they think a question should be closed or open, should it come to that. – TylerH May 13 at 14:10
  • But in small sites re-opening basically doesn't happen at all. Furthermore, I totally disagree that questions should be closed and re-opened and closed again like that... it's not good for the (sometimes new) users to have to see their question's status constantly changing like that. You say "everyone with the rep requirement gets an equal opportunity to say whether they think a question should be closed or open", which is already the case whether the requirement to close is 3 votes or 10 votes. I just don't think on a new (and small) site we should be closing questions with only 3 votes. – user1271772 May 13 at 19:10
5

I don't think that a question should be closed or opened by three "random" votes. On the other hand, we could "get there" in a roundabout way by giving certain individuals "double" votes.

This would be a user privilege, whereby users with 10,000 rep (or some other threshold) could effectively cast TWO close or reopen votes instead of one. THREE such users would cast six votes. Or it could be two double voters and one single voter, or one double voter and three single voters or the current five single voters to add up to a total of five.

That way, we would retain the "fabric" of five close or re open votes, while allowing close or reopening by three "senior" users instead of five users or one moderator.

Another possibility is to allow moderators to choose to cast a "double" vote instead of "five" votes. Many moderators shy away from unilaterally taking "five" vote actions in order to defer to the community, but would welcome the chance to "personally" cast single votes, and probably would welcome a chance to cast a double (multiple) vote that recognizes their seniority. Of course, they would retain the right to cast "unilateral" votes in the case of spam or bad (not just mediocre) content.

Edit: New sites in their "start up" period would have reduced thresholds for "double votes" on closing and reopening, just as "single" votes now do.

4

I'm going to interpret this as a formal review request that has been actually actioned on my A&M Meta question.

I'm glad to see that I received an answer, and I'm happy also to see that this is being expanded in the relatively sensible fashion that has been laid out.

I'm bitter behind the fact that I had to wait over a year to hear anything about this, to the degree that I'm concerned that the efficacy of this test will falter due to lack of participants, which can only undermine both the value of the feature and the data in the test.

1
  • 4
    As I've now answered your meta question (it was always my intention to do so, just needed a minute to do so) I hope it helps shed some light on both the circumstances and my own personal struggles. :) I've already learned a lot about what data I should look at just from having this question here, so I really appreciate the insights people are giving me and the time people are taking to help. – Catija May 5 at 23:17
3

After the 45 day test period, the system will return to [at least until policy is permanently changed] requiring 5 votes being needed to close/reopen.

Question: will posts closed with only 3 votes during the 45 day test window reopen after 45 days when the threshold returns to 5 votes? Or will they remain closed?

4
  • 10
    They stay closed. – Catija May 7 at 18:21
  • I think the answer would be relatively easy to guess if one asked "symmetric" question: will posts reopened with only 3 votes during the 45 day test window re-close after 45 days when the threshold returns to 5 votes – gnat May 7 at 19:08
  • 5
    @gnat I disagree that it'd be easy to guess. I've spent too much time on this earth to continue assuming the sensible is always true. I think this is a fair question, and I'm glad to find out that the sensible answer was in fact true. – theforestecologist May 7 at 19:52
  • 1
    I agree, SE got many quirks that go against common sense, so it's really not trivial to ask such thing. Well at least this one isn't such a quirk. :-) – Shadow Wizard Wearing Mask V2 May 8 at 9:05
3

This seems to boil down to "high-rep users are not toiling hard enough in the moderation queues." To me this means incentives are misaligned. I know that badges are the current carrot to induce users to do this work, but I also know that badges aren't enough to get me to review very many posts.

Why would a high-rep user want to deal with a stream of generally awful posts every day? What would make it more bearable? What would make it fun? In the past you could at least have the satisfaction of informing a homework poster that they are a moron, but we are told to be nice to new users now. :-) But after composing a thoughtful and patient answer to a question only to see it read and immediately deleted so that the homework poster can get away with cheating, a high-rep user is in no mood to deal with more of the same in the moderation queues.

So. Figure out a way to make cleaning the Augean stables stimulating and you won't have any problem finding five votes to close or reopen posts. My wife's foreign language course uses literal bells and whistles to encourage her to keep plugging away at her lessons. I don't know if that's the right carrot for this site, but it is clear that the virtual pay is too low.

1
  • 1
    eh, anything that reduces the number of such posts people need to look at to result in the site being "cleaner" will make the queues less of a chore, IMO. If you feel like your actions are actually having an impact, it feels better, even if in the end you hit the same limit each day. – Kevin B May 21 at 3:29
2

It might be very interesting to compare SE with its antithesis, USENET. It's hard to earn a "forged cancel" there at all (you must be a hardboiled spammer), and in theory even crackpots flooding a group can be silenced by killfiling, i.e. the work needed is on the casual user, not a mod.

Thus, fundamentals shall be considered first: How many low quality remaining on a SE will hurt it in the long run (in contrast to, say, them simply being ignored)? With a high inflow of good questions, they evidently seem less a problem to me. (For a few random examples of different categories I know, sci.math died the crackpot death, the German SF group is clean but lacks interest, the German politics groups are a cesspool, but the German joke group kinda survives up to now, conflicting interests declared.)

I clearly can relate to SE officials seeing the downfall of USENET and prefer to err on the side of caution in return. More conflicting interests declared: If I enter a group and get my first post nuked for being unclear, it's rather improbable that I try to reword it into clearness and blame the thematic itself to be unclear. (Questions are brain-children, insult them and you insult the parent :-)

To sum up my personal stance on the matter: If in Rome etc. - I'm just a customer here.

5
  • 2
    at SE, most work needed is also on casual user, see eg A Theory of Moderation. The very change proposed here is intended to shift even more of the work to these users. (also, a minor nitpick, you are not a customer - first, because you don't pay for service, and most importantly, because folks providing content at SE are unpaid volunteers) – gnat May 6 at 9:57
  • @gnat: THX for the link, will read later. (And "customer" was the best approximation I could come up with :-) – Hauke Reddmann May 6 at 12:54
  • 5
    Posts don't get nuked instantaniously on SE/SO. They get closed, yes. They might get downvoted too. But if you edit your post according to the guidance given in the close notice you have a good chance of redeeming yourself. – Luuklag May 6 at 13:15
  • 4
    You are not a customer here (your last sentence states that you are), you are a user, as are most (including me), so there is no way to question a "seller" for an unsuccessful business or a bad product, SE network sites aim to add quality content that should be useful for future visitors and not just the interest of a single individual who arrives with a question with a "giant code" or that we cannot easily reproduce, we should not be a helpdesk, this is not a "trade", it is a "community". I hope you understand how constructive the criticism is. – Guilherme Nascimento May 6 at 20:37
  • @GuilhermeNascimento: I can't count how often I could recycle an useful answer (mainly coding SEs), and I love SE for that, so of course you are right. (I doubt, though, that my questions will have that fate, as they tend to be rather specific. OK, on Chess I already got a Nice Question Badge, so not all hope is lost :-) – Hauke Reddmann May 7 at 7:51
1

How can a site be nominated for this feature? I asked at our meta if people agree with the request. Should moderators ask it to the staff?

By the way, at Earth Science what we find difficult is to delete questions, much more than closing them. Is it possible to reduce the number of deleting votes as well?

1
  • 7
    You should have a discussion on your site's meta to see if the community would support lowering the threshold for close-votes. If that is true, you should have a mod tag that question [status-review], that will signal a CM to come have a look. – Luuklag May 10 at 12:43
0

Are there Stack Exchange sites where 3 votes are sufficient for closure but 5 are required for reopening? I seem to have observed this on Chess.SE. What's the reason for this asymmetry?

4
  • 4
    There is no asymmetry. The system requires the same number of votes to close and reopen. There are specific people (like diamond moderators) whose votes can be unilateral, but no sites have mismatched numbers of votes required to close and reopen. – Catija Jun 18 at 14:03
  • 1
    @Catija One thing that did come up on EL&U's meta was that votes on the trial site's metas are still at 5 instead of 3. Maybe that explains why someone might misremember needing more votes to reopen than close. – ColleenV Jun 18 at 14:23
  • 4
    @ColleenV It's possible but there's a specific case on Chess that this user was involved in where the question was closed after three votes, where the mod was the third vote, and the question now has three reopen votes but hasn't been reopened (because this test is not active on Chess), so that's likely what this is in reference to. – Catija Jun 18 at 15:39
  • 1
    @Catija Thanks. I was confused about how that happened. – Mobeus Zoom Jun 18 at 15:42
-2

Will site moderators have the option of choosing for their own site 3 vs 5?

I think this is a great idea, but I would like to ask if the feature will be hardwired, or if moderators will have the ability to choose 3 vs 5 votes? In an ideal world, participation is encouraged and increased such that going back to 5 votes makes sense. After all, we're not just trying to block poor questions before they get too involved, we're also trying to train our stack users to become involved in the culture of the site, both by processing reviews and by participating in the meta discussions that lead to cultural refinement.

If this proves successful, do we hardwire all of Stack Exchange to 3 votes, regardless of any stack's ability to educate and motivate their user bases? Or will the mods have the ability to switch back and forth as they see fit to accommodate the amount of support they're receiving?

3
  • 7
    I've already answered this twice - I know there's lots of answers, but please read them. We will not change this without the buy-in of the sites! We have no intention of forcing this network-wide. – Catija May 11 at 1:53
  • 4
    @Catija You're busy, so am I. I read the answers and I got the fact that you're implementing this as a one-off change if stacks request it. But that wasn't what I asked. If this is a one-off (which you mentioned in a comment to user1271772's post), then this appears to be a use-it-or-lose it permanent solution. And I'm specifically asking if that's actually the case or not. Can stacks move back and forth at will or not? If not... think of this as a frustrating enhancement request. – JBH May 11 at 2:17
  • 8
    There's no requirement that it be permanent forever but changing between them need to thought out and with community discussion. Mods will always be able to bring those requests to the CM's attention and we'll pull data and recommend what we think would be helpful. In general, I don't think there's much likelihood that sites will need to change this without there being a huge change in how closing works, the activity of reviewers, etc. – Catija May 11 at 2:22
-20

Personally, I would like to see the the vote count be higher on sites like Stack Overflow. I have had (and have seen) many questions immediately shot down due to easy to fix things like missing information or a lack of understanding. I feel like this is a waste of potential, as (in my eyes at least), many of those questions could have had a good outcome had people not been so quick to close (or, in this case, had been harder to close such as needing more votes). Just my two pennies' worth.

14
  • 28
    If it is easy to fix then the OP has all the chance to do so. Either after the first close vote came in or after closing. You do know questions can get re-opened, right? – rene May 4 at 6:48
  • 6
    It seems you haven't known about the never-ending Close review queue problem on Stack Overflow... – Meta Andrew T. May 4 at 6:48
  • 10
    We have been asking for years to lower the cv threshold: meta.stackoverflow.com/questions/391832/… let's at least wait for another 6 to 8 years before reversing that decision. – rene May 4 at 6:51
  • 6
    Please see How long should we wait for a poster to clarify a question before closing?. As an OP, the appropriate response to a closed question is to edit the question to provide the missing details or focus or clarify the question. Then it can be reopened. – Sebastian Simon May 4 at 8:18
  • 10
    @EkadhSingh it doesn't need a re-open review. You can re-open vote on any closed question. The lack of guidance how to proceed when a question gets closed is a problem. No one seems to be able to find the info we have on that but strangely enough they do find the whine button under their posts, on Meta, chat and Twitter. And during all those complaints not a single character is improved on the closed question. Such a shame. You're accepted as my advocate. I need one. – rene May 4 at 13:15
  • 1
    @rene that maybe so (that questions can get re-opened outside of the queue), but most closed questions that are edited never get reopened, see this SEDE query which shows that that statistic is around 15%. Obviously, most edits will be trivial, but I doubt (have no evidence to back this up though) that only 15% of edits are nontrivial. – Ekadh Singh May 4 at 14:14
  • 4
    @EkadhSingh I'm aware that the SO scale of things don't help in effectively curating posts that needs curating and I appoligize for my laziness. But making it harder to close vote stuff while my gut says that 85% of new questions (on SO) should be closed is not something I can support. I would love that all curating and all review queues work and are effective. it should, given the premise that the SE Q/A model wants to be better then a forum. We only need a couple of thousand voters / reviewers each day to get that back on-track. – rene May 4 at 14:32
  • 1
    @rene I know, I never said that questions should be harder to close, in fact I completely agree with the changes Catija made, I was just playing devils advocate. As far as I can tell, the new changes will (hopefully) make good questions get re-opened again too more often too, so I see this as an absolute win. Just like playing devils advocate :) – Ekadh Singh May 4 at 14:37
  • When I said easy to fix, I meant for newer users, who don't really have an idea of what to fix. Obviously, now I know how to edit and have questions reopened, but I was also a new user at one point, who knew nothing about editing or reopening questions. All I'm saying is, people are a bit too vicious when it comes to closing posts, and from experience, that can really deter a new user away from the site(s). – Werlious May 4 at 19:28
  • I suspect "further" flexible vote thresholds would be a seperate meta question. Also close and reopen votes are tied together so a 'harder' closure also means more difficult reopening – Journeyman Geek May 5 at 0:26
  • 2
    So do you want the site to be a helpdesk with questions asked anyway to meet only the need of the OP rather than a knowledge repository with elaborate questions that can serve different people with similar problems? – Guilherme Nascimento May 5 at 3:32
  • @GuilhermeNascimento All they appear to be saying is that they are wary of making closing easier when it seems there's not a robust procedure in place to help Askers improve their Questions. Askers improving their Questions helps with both goals you mention. – trlkly May 5 at 4:46
  • 10
    @trlkly I disagree with turning the site into a "helpdesk" to solve problems that are only pertinent to the OP, the questions have to serve as a "knowledge repository" (good questions that serve other people as well). – Guilherme Nascimento May 5 at 4:54
  • I believe all this is a good analogue of the general attitude on the stack network – Werlious Jun 3 at 18:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .