36

In the Community Roadmap for Q4 of 2020, one of the goals for November 2020 is

Three-vote Close and Reopen Tests

Three-vote closure has been requested by many many sites, including the one I moderate. In many cases, the reason for the request is that the community is smaller than that of Stack Overflow, and the number of users that are active in the close-vote review queues makes 5-vote closure too slow.

As November 2020 has already passed, and motivated by inquiries from members of Spanish.SE, I want to ask here: what is the status on this? Do we expect the tests to start soon (or have they already started on some SE site)? Or has this been delayed?

2
  • 2
    I think they're busy planning what they need to plan in order to be able to give answer. – Shadow The Vaccinated Wizard Dec 11 '20 at 15:23
  • That probably fell through the cracks. – Trilarion Dec 11 '20 at 16:38
22

Update

As of 3 May 2021 the tests are scheduled to begin on 6 May on 13 sites.


We're working on it. Nicolas and I are both assigned to this project to investigate the benefits of this change. We got a late start because there was a mix-up in having this project worked on in November but we're working on it. Here's our (general) plan:

Collect data & consider various complicating factors

The first step is to mature the SEDE queries we're already using and to look at other factors that may explain low participation in review. As with the tests I ran on sites a year ago, there are two main factors we are looking at:

  1. Review completion rate - For any post that goes into review, what percentage actually get a verdict (e.g. close or leave open) vs what percentage age out of review. In general, if a high percentage are being completed, the site is likely doing just fine with five reviewers. If the completion rate is low, then it's likely that reviews are not getting done for some reason. This includes defining what a "low" completion rate is.
  2. Moderator vs. User participation - One of the reasons a completion rate may be high is because the site moderators are doing a bulk of the reviews that other users aren't. This can put an undue burden on the moderators and makes reopening harder as the mods will also have to review and track the reopen queue.

In the case of moderators reviewing and closing votes, we're expanding our research to also look at which vote mods are casting. If they're largely casting the third or fourth vote, then a three-vote closure will likely be more impactful and reduce the mods' work load but if they're casting early votes, like the first or second, that could indicate a few things - there aren't sufficient people available and active in review or, if there are, perhaps moderators might need to consider letting users handle review a bit more than they currently are. In both cases, three vote closure may still be beneficial!

One of the things we're not going to be looking at directly will be whether those reviews were accurate or not. The reality is, we simply aren't able to judge whether a question should be closed or not - on all of our 170+ sites. For the purpose of deciding where to test this feature, we're assuming that reviews are accurate.

There's an additional factor that I want to consider for this test and it relates to the review queue indicator. This is the dot that appears on the review icon to let users with sufficient reputation know that there's something that needs their attention. In my experience, there are some sites where there simply aren't enough review tasks in the queue at any given time to ever light up, which requires someone seek out items to review rather being asked to help. So, on sites where there seem to be sufficient reviewers, we may look at changing when that indicator turns on rather than (or in addition to) changing the number of votes to close. We do also need to be careful not to make the indicator so noisy that people simply ignore it!

As part of this step we'll also be creating success and failure metrics similar to the ones we've used before - things like whether the completion rate increases and/or the moderator participation rate decreases. We also want to avoid close/reopen warring.

Identifying sites and running the tests

Once we have our data pulled together, we'll identify 10-12 sites that fall into the groups that we think would most benefit from these changes and we'll focus on the sites that have already requested it - but we may also consider sites that may not have requested but could benefit from the test. The plan is to start the tests in January and run for about 45 days. Since many sites experience activity downturns during December, we're waiting a bit to get past that so that we can get good data.

After the testing period, we'll reset the votes needed to five while we review the data we get. I also like getting specific feedback from the users on these sites through a meta post.

Reviewing data and next steps

Once we have the tests completed, we'll look at whether we found the tests to be successful or not and identify what factors seemed to indicate better success rates. If we feel the tests are successful, we'll make the change to 3 vote closure permanent along with any changes to the review queue indicator.

Once we feel like we can predict how these changes will impact sites, we'll identify more sites that look like they'd also be successful and are open to this change. While we'll check in on the impact of the change on these sites, we're not planning to run it as a test, since we're hoping the tests on other sites should be enough.


These plans are still flexible as we're still making them, so if you have thoughts and ideas, please let me know.

17
  • Looks nice! Let's see what we learn. "there are some sites where there simply aren't enough review tasks in the queue at any given time to ever light up [...] where there seem to be sufficient reviewers" aren't those the sites that do not need this in the first place (i.e., the sites that won't get selected for testing anyway)? Is there any other motivation for three-vote closing than the one I mention in the question (queues filling up/reviews taking too long)? – wimi Dec 17 '20 at 16:26
  • 1
    They could still get selected, sure - this isn't necessarily about bulk! If a site has only 20 questions that need closure in a month but only 5 of them get reviewed, that's only a 25% completion rate, so we still want to investigate why that percentage is so low and help improve that - if someone on the site thinks the question should be closed and there just aren't enough people to close it, this change is still valuable for them. :) – Catija Dec 17 '20 at 16:28
  • 2
    For the last part: When identifying sites that "look like they'd also be successful and are open to this change", will SE reach out in any way to the site or it's moderators to see if the site is open to this change, or is it expected that there's already an existing meta discussion on lowering the close vote threshold somewhere? – Tinkeringbell Dec 17 '20 at 17:26
  • 3
    @Tinkeringbell We realize that some sites may not be aware of this option and may really benefit from it - for that reason, we won't be relying on meta discussions only. If we think a site needs the change based on our data, we'll reach out to the mods to see if they want to start a discussion. :) – Catija Dec 17 '20 at 17:27
  • 5
    On the data front -- one of the reasons we wanted to try this on Physics.SE is the way we treat homework questions. Those tend to get closed eventually, but not before they accumulate answers in violation of our policies. One suggestion for data queries would be to allow a breakdown by tag (which would let us see specifically how homework looks) and another suggestion would be to look at number of answers posted for each close vote count... For example, if a Q gets to 3 votes before it gets any answers, but then answers come in before it gets to 5, then closing at 3 votes is a benefit – tpg2114 Dec 17 '20 at 17:57
  • 1
    @Catija tpg beat me to it, but I'll add another voice: please add, as a measure of closure effectiveness, the fraction of questions that get answers before they get closed. Regardless of the tag, that is an indication of site policies not working correctly. – E.P. Dec 18 '20 at 19:17
  • @E.P. Or an indication of people not reading the policy. It's not that simple. – Mast Dec 18 '20 at 19:29
  • @Mast If people are not reading the policy then it's not working as intended, either way. The core point is that question closure is always meant to prevent questions from getting answers (which for various reasons the site in question has decided would be a bad thing). If the questions are getting those answers before the closure process has concluded, it's an indication that it's not working fast enough. – E.P. Dec 18 '20 at 19:36
  • 1
    Regarding Physics - I've been thinking about y'all a lot... The thing is, the focus of this test, at least initially, is on sites where either stuff just isn't getting reviewed at all or it's largely being done by mods. I feel your struggles when it comes to wanting to close stuff sooner but that feels like two problems in one and I don't know that faster closes would help. Also, based on some queries Shog shared, I'm pretty sure that the answers are coming even before you get three votes, so I don't know this will help y'all much. – Catija Dec 18 '20 at 20:14
  • 1
    If I'm reading the data I have correctly, for questions closed with five votes over the last 90 days (on Physics), the first answer, on average, comes at 1.67 votes... meaning it comes between the first and second vote (on average)... this means that, even with three votes to close... over half of the questions would still be answered. @tpg2114 – Catija Dec 18 '20 at 20:17
  • 1
    @Catija That's an interesting bit of info that we wouldn't normally have insight into, thanks! I'm a bit biased certainly so I'd like to be in the initial rounds of testing, but I understand if we're not. We do have a really good group of folks who try to keep on top of things so the mods don't have to do as much closing. I think when I originally pitched it to you or Shog, it was to see how a medium-sized site is impacted (SO was a large size test, and there's plenty of smaller sites, but we're near the middle of the pack) and that might not be an immediately useful test anymore. – tpg2114 Dec 18 '20 at 21:43
  • 1
    @tpg2114 Y'all aren't in the middle, when it comes to closing - for the 90 day period I checked, y'all are seventh in number of posts nominated for closure. And, of those sites, you're second best when it comes to actually getting through those - 78% handled. :) So that's great! I have a longer answer written for your Meta but I think there's something wrong with one of the queries, so I'm seeing if Taryn can look at it for me. – Catija Dec 18 '20 at 21:45
  • 3
    Bearing in mind that it hasn't been more than 6 to 8 Weeks, but is there any update on this matter? Have the tests been completed? – Sam Onela Feb 10 at 16:24
  • 2
    Any news on this? Did the initial tests take place? – wimi Apr 7 at 6:44
  • 1
    @Wimi if you haven't already - check out Testing three-vote close and reopen on 12 network sites – Sam Onela May 3 at 20:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .