This post details the second in a series of releases related to the ongoing work on Review Queues. It focuses on big upgrades to the tools provided for moderators to track and apply review suspensions to users. These changes are now live network-wide.
The Current Interface
In order to appreciate the magnitude of changes being released here, it will be helpful to show what the interface looks like today (and has looked like for quite some time). The tools currently consist of two pages: one that shows current review suspensions and allows for adding new review suspensions, and one showing historical suspensions (click on the links to view screenshots). The interface for suspending a user is pretty barebones, allowing a mod to search by user ID, set suspension duration, and enter the suspension message:
Over time, some mods (special shout out to Stack Overflow moderator Samuel Liew) have created and maintained a set of user scripts to add more functionality to this interface. However, the lack of functionality and options in this section is something that has placed an unfair burden on moderators who sought to improve the quality of reviews through review suspension enforcement.
Work defining the new functionality was led by our Product Designer Lisa Park who spent a good deal of time reviewing years’ worth of feedback, interviewing a number of moderators who are active with review suspensions, and going through several rounds of design and prototyping. In addition to modernizing the design with Stacks, we sought to address weak points with the usability of the current interface, introduce innovations that have been included in moderator user scripts, and add more options for message templating and review action selection (more on that below).
New UI for Creating Review Suspensions
This now breaks down the creation of review suspensions into a multi-step process.
Improved user search and suspension accessibility
Access to the tool is available to mods through their
admin section, or from direct links in the user's profile or review tasks. When on the page, Moderators can begin by searching for the user ID or profile URL of the user in question. This will check for review suspension eligibility (the following cannot receive a review suspension: moderator, users with existing review suspensions, and users who have never performed a review).
Ability to easily cite problematic reviews
When an eligible user is loaded, their profile is shown (as well as the number of times that the user was suspended from reviews in the past, along with a link to show details on this), as well as a list of their 15 most recent reviews.
This table can be toggled using the drop down to show the last 15 tasks for the user in all review queues that are active on the site, as well as the last 15 audits (all audits and failed audits).
The reviews shown in this table can be selected for inclusion in the review suspension message using the [Add task] button. Reviews not included in the table can be added directly using the text box above the table.
Customizable message templates
We’re including options for 11 templates which were based in large part on templates that had already been in use by moderators, as well as feedback we received through other avenues. We also shared drafts of these messages with moderators on their Stack Overflow Team for a final round of revisions. The template messages can be previewed in the modal.
After clicking [Use this template], the moderator is shown the actual review suspension message that has been generated using the selected review tasks combined with the review suspension message templates. Below a set of review suspension message guidelines, the generated message shows up in a text editor, which is fully editable by the moderator.
A duration for the suspension must be selected at this point. These will default to the escalation rules that have been in place for automated audit-based bans since 2016 (start at 2 days, double the previous length whenever a new automatic suspension is started within 30 days of the end of a previous suspension, half the previous length if not).
Once the review suspension has started, the user will see the suspension message the next time that they access review queues. Messages are now rendered including line breaks and lists.
Current & Historical Suspensions
The screen showing current suspensions includes the same columns as before, using our new design palette, with more sorting and filtering options. The Reason column displays the message template that was used, which can be clicked to toggle display of the actual message:
Review suspensions given based on automation relaing to failed audit reviews are now clearly highlighted as such, and the suspension messages for these are generated using the new Robo-reviewing template.
New Help Center Articles
We have expanded the body of content available in the help center related to review queues. The goal is to provide reviewers with more education about how review queues work, and to have a canonical place to point them to when they get stuck or suspended. Queue-specific suspension messages will link directly to the Help Center article that is dedicated to that queue.
Metrics & Feedback
The metrics that we will be tracking to judge the overall success of these upgrades and new features will include:
Repeat reviewer suspensions and changes in duration: we hope that both these numbers will go down as improvements in product education and the quality of moderator guidance goes up.
Number of unique moderators creating review suspensions:
The old interface presented a barrier to entry for moderators to get involved with review suspensions and to create quality messages
New entry points to review suspensions make the feature easier to find and more efficient to access
Direct feedback from moderators about review suspensions
Visits to the suspension page, and conversion rate of visits to page through to actual suspensions
We are excited to be able to provide this new functionality to the moderator community, and are hoping that these features will contribute positively to the efforts of moderators who take an interest in reviews and content quality. A more detailed version of this post has already been shared with the moderator community and we are collecting feedback and bug reports directly from them. We are sharing it here in order to keep the overall Stack Exchange Community informed about changes to important site operations. We welcome your thoughts below.