New Review Suspensions Mod UX

Question

This post details the second in a series of releases related to the ongoing work on Review Queues. It focuses on big upgrades to the tools provided for moderators to track and apply review suspensions to users. These changes are now live network-wide.

The Current Interface

In order to appreciate the magnitude of changes being released here, it will be helpful to show what the interface looks like today (and has looked like for quite some time). The tools currently consist of two pages: one that shows current review suspensions and allows for adding new review suspensions, and one showing historical suspensions (click on the links to view screenshots). The interface for suspending a user is pretty barebones, allowing a mod to search by user ID, set suspension duration, and enter the suspension message:

Over time, some mods (special shout out to Stack Overflow moderator Samuel Liew) have created and maintained a set of user scripts to add more functionality to this interface. However, the lack of functionality and options in this section is something that has placed an unfair burden on moderators who sought to improve the quality of reviews through review suspension enforcement.

Work defining the new functionality was led by our Product Designer Lisa Park who spent a good deal of time reviewing years’ worth of feedback, interviewing a number of moderators who are active with review suspensions, and going through several rounds of design and prototyping. In addition to modernizing the design with Stacks, we sought to address weak points with the usability of the current interface, introduce innovations that have been included in moderator user scripts, and add more options for message templating and review action selection (more on that below).

New UI for Creating Review Suspensions

This now breaks down the creation of review suspensions into a multi-step process.

Improved user search and suspension accessibility

Access to the tool is available to mods through their admin section, or from direct links in the user's profile or review tasks. When on the page, Moderators can begin by searching for the user ID or profile URL of the user in question. This will check for review suspension eligibility (the following cannot receive a review suspension: moderator, users with existing review suspensions, and users who have never performed a review).

Ability to easily cite problematic reviews

When an eligible user is loaded, their profile is shown (as well as the number of times that the user was suspended from reviews in the past, along with a link to show details on this), as well as a list of their 15 most recent reviews.

This table can be toggled using the drop down to show the last 15 tasks for the user in all review queues that are active on the site, as well as the last 15 audits (all audits and failed audits).

The reviews shown in this table can be selected for inclusion in the review suspension message using the [Add task] button. Reviews not included in the table can be added directly using the text box above the table.

Customizable message templates

We’re including options for 11 templates which were based in large part on templates that had already been in use by moderators, as well as feedback we received through other avenues. We also shared drafts of these messages with moderators on their Stack Overflow Team for a final round of revisions. The template messages can be previewed in the modal.

After clicking [Use this template], the moderator is shown the actual review suspension message that has been generated using the selected review tasks combined with the review suspension message templates. Below a set of review suspension message guidelines, the generated message shows up in a text editor, which is fully editable by the moderator.

A duration for the suspension must be selected at this point. These will default to the escalation rules that have been in place for automated audit-based bans since 2016 (start at 2 days, double the previous length whenever a new automatic suspension is started within 30 days of the end of a previous suspension, half the previous length if not).

Once the review suspension has started, the user will see the suspension message the next time that they access review queues. Messages are now rendered including line breaks and lists.

Current & Historical Suspensions

The screen showing current suspensions includes the same columns as before, using our new design palette, with more sorting and filtering options. The Reason column displays the message template that was used, which can be clicked to toggle display of the actual message:

Review suspensions given based on automation relaing to failed audit reviews are now clearly highlighted as such, and the suspension messages for these are generated using the new Robo-reviewing template.

New Help Center Articles

We have expanded the body of content available in the help center related to review queues. The goal is to provide reviewers with more education about how review queues work, and to have a canonical place to point them to when they get stuck or suspended. Queue-specific suspension messages will link directly to the Help Center article that is dedicated to that queue.

Metrics & Feedback

The metrics that we will be tracking to judge the overall success of these upgrades and new features will include:

• Repeat reviewer suspensions and changes in duration: we hope that both these numbers will go down as improvements in product education and the quality of moderator guidance goes up.

• Number of unique moderators creating review suspensions:

  • The old interface presented a barrier to entry for moderators to get involved with review suspensions and to create quality messages

  • New entry points to review suspensions make the feature easier to find and more efficient to access

• Direct feedback from moderators about review suspensions

• Visits to the suspension page, and conversion rate of visits to page through to actual suspensions

We are excited to be able to provide this new functionality to the moderator community, and are hoping that these features will contribute positively to the efforts of moderators who take an interest in reviews and content quality. A more detailed version of this post has already been shared with the moderator community and we are collecting feedback and bug reports directly from them. We are sharing it here in order to keep the overall Stack Exchange Community informed about changes to important site operations. We welcome your thoughts below.

Answer 1

status-planned

So my only suggestion would be to provide an interface similar to that of reputation that’s only accessible to the user.

So for transparency reasons, how difficult would it be, to provide a page for the user that indicates which audits they have failed? This page could also display when they were review suspended. This might help users realize that they have been suspended in the past.

It seems users are review suspended several times without actually realizing it.

Answer 2

status-completed

The Current suspensions table now clearly shows which ones were automated suspensions. In these cases, we now use the Robo-reviewing suspension template (which will also highlight the culprit failed audit reviews for the user)

Make it easier to tell when automatic suspensions were imposed for failing audits, and which failed audits were the cause of the suspension

The pages, as described here, do a great job at informing moderators of review audits a user failed as well as indicating when review suspensions were imposed for failing audits. But those two things are listed on two separate pages, which means it's harder to piece together the overall history of the user failing audits and how they led to a suspension.

Both of these items should be listed on the same page, so that it's easier for moderators to tell when and why a user was suspended for failing audits. Ideally, this would be in the review suspension history page: for automatic suspension entries, it should indicate the specific failed audits that led to that particular suspension (the three failed in the last 30 days, or the one failed within 30 days of coming off a previous suspension, per the current way those are imposed).

This way, if a user comes to meta and complains about being review-suspended, a moderator can point them to the specific review audit(s) that led to the suspension, which the user may have forgotten about.

Also, I've often seen cases where moderators are unaware of the rule that a single audit failure within 30 days of coming off a prior review suspension will lead to an immediate re-suspension. Most of the time, users come to meta sites complaining that they were suspended for a single failure, and moderators speak to them in a way that implies that they're lying ("suspensions aren't given for single failures", etc.). While that's technically true if you count the previous audit failures that led to the first suspension, from what I can tell, mods sometimes aren't aware of this rule.

And in the case where that single failed audit was actually a bad audit, it's hard for mods to go through and piece all the history and find that it was caused by that single failure.

Answer 3

status-completed

These links no longer show next to the review records of moderators

When looking at my own review (or one of my moderator colleagues), there's a link to 'suspend user from review'.

However, this is not possible, as the next page shows:

Just like you can't send a mod message to your own account, can the link be deactivated or even grayed out?

Answer 4

The filter for the type of review in Step 2 lists review types that might not exist on the site. For example, the site I moderate, Arqade, only has the Close Votes, Low Quality Posts, First Posts, Late Answers, Reopen Votes, and Suggested Edits review queues. As shown in the screenshot below, however, the filter in the new review suspension UI shows other review types, which of course don't return any review actions when selected.

Could these be removed?

---

This answer was marked as completed, but the issue still remains for the "All audits" and "Failed audits" review types, as shown below.

I believe this is not intended behaviour.

Answer 5

Is suggesting the "Skip" option in the default review suspension messages appropriate behaviour?

Two review suspension messages mention this:

Although the post reviewed was posted as an answer, it did not attempt to provide an answer to the question. Please review the question to see if the answer is attempting to provide a solution. Flagging it as Not an answer so that the post(s) could be removed and then using the Skip action would have been more appropriate.

and

Although it was posted as an answer, it contained nothing more than a link to an off-site resource and that does not meet our minimum standards for an answer. Flagging it as Not an answer or Very low quality so that the post could be removed and then using the Skip on the review would have been more appropriate.

Surely we should encourage reviewers to use the "I'm Done" action after reviewing a post, right?