Update
As of 3 May 2021 the tests are scheduled to begin on 6 May on 13 sites.
We're working on it. Nicolas and I are both assigned to this project to investigate the benefits of this change. We got a late start because there was a mix-up in having this project worked on in November but we're working on it. Here's our (general) plan:
Collect data & consider various complicating factors
The first step is to mature the SEDE queries we're already using and to look at other factors that may explain low participation in review. As with the tests I ran on sites a year ago, there are two main factors we are looking at:
- Review completion rate - For any post that goes into review, what percentage actually get a verdict (e.g. close or leave open) vs what percentage age out of review. In general, if a high percentage are being completed, the site is likely doing just fine with five reviewers. If the completion rate is low, then it's likely that reviews are not getting done for some reason. This includes defining what a "low" completion rate is.
- Moderator vs. User participation - One of the reasons a completion rate may be high is because the site moderators are doing a bulk of the reviews that other users aren't. This can put an undue burden on the moderators and makes reopening harder as the mods will also have to review and track the reopen queue.
In the case of moderators reviewing and closing votes, we're expanding our research to also look at which vote mods are casting. If they're largely casting the third or fourth vote, then a three-vote closure will likely be more impactful and reduce the mods' work load but if they're casting early votes, like the first or second, that could indicate a few things - there aren't sufficient people available and active in review or, if there are, perhaps moderators might need to consider letting users handle review a bit more than they currently are. In both cases, three vote closure may still be beneficial!
One of the things we're not going to be looking at directly will be whether those reviews were accurate or not. The reality is, we simply aren't able to judge whether a question should be closed or not - on all of our 170+ sites. For the purpose of deciding where to test this feature, we're assuming that reviews are accurate.
There's an additional factor that I want to consider for this test and it relates to the review queue indicator. This is the dot that appears on the review icon to let users with sufficient reputation know that there's something that needs their attention. In my experience, there are some sites where there simply aren't enough review tasks in the queue at any given time to ever light up, which requires someone seek out items to review rather being asked to help. So, on sites where there seem to be sufficient reviewers, we may look at changing when that indicator turns on rather than (or in addition to) changing the number of votes to close. We do also need to be careful not to make the indicator so noisy that people simply ignore it!
As part of this step we'll also be creating success and failure metrics similar to the ones we've used before - things like whether the completion rate increases and/or the moderator participation rate decreases. We also want to avoid close/reopen warring.
Identifying sites and running the tests
Once we have our data pulled together, we'll identify 10-12 sites that fall into the groups that we think would most benefit from these changes and we'll focus on the sites that have already requested it - but we may also consider sites that may not have requested but could benefit from the test. The plan is to start the tests in January and run for about 45 days. Since many sites experience activity downturns during December, we're waiting a bit to get past that so that we can get good data.
After the testing period, we'll reset the votes needed to five while we review the data we get. I also like getting specific feedback from the users on these sites through a meta post.
Reviewing data and next steps
Once we have the tests completed, we'll look at whether we found the tests to be successful or not and identify what factors seemed to indicate better success rates. If we feel the tests are successful, we'll make the change to 3 vote closure permanent along with any changes to the review queue indicator.
Once we feel like we can predict how these changes will impact sites, we'll identify more sites that look like they'd also be successful and are open to this change. While we'll check in on the impact of the change on these sites, we're not planning to run it as a test, since we're hoping the tests on other sites should be enough.
These plans are still flexible as we're still making them, so if you have thoughts and ideas, please let me know.
