82

This post details the second in a series of releases related to the ongoing work on Review Queues. It focuses on big upgrades to the tools provided for moderators to track and apply review suspensions to users. These changes are now live network-wide.

The Current Interface

In order to appreciate the magnitude of changes being released here, it will be helpful to show what the interface looks like today (and has looked like for quite some time). The tools currently consist of two pages: one that shows current review suspensions and allows for adding new review suspensions, and one showing historical suspensions (click on the links to view screenshots). The interface for suspending a user is pretty barebones, allowing a mod to search by user ID, set suspension duration, and enter the suspension message:

Old user interface for suspending users from reviews. Shows textbox for entering a User ID, multi-line textbox for entering a review suspension message, and radio buttons for setting review duration

Over time, some mods (special shout out to Stack Overflow moderator Samuel Liew) have created and maintained a set of user scripts to add more functionality to this interface. However, the lack of functionality and options in this section is something that has placed an unfair burden on moderators who sought to improve the quality of reviews through review suspension enforcement.

Work defining the new functionality was led by our Product Designer Lisa Park who spent a good deal of time reviewing years’ worth of feedback, interviewing a number of moderators who are active with review suspensions, and going through several rounds of design and prototyping. In addition to modernizing the design with Stacks, we sought to address weak points with the usability of the current interface, introduce innovations that have been included in moderator user scripts, and add more options for message templating and review action selection (more on that below).

New UI for Creating Review Suspensions

This now breaks down the creation of review suspensions into a multi-step process.

Improved user search and suspension accessibility

Access to the tool is available to mods through their admin section, or from direct links in the user's profile or review tasks. When on the page, Moderators can begin by searching for the user ID or profile URL of the user in question. This will check for review suspension eligibility (the following cannot receive a review suspension: moderator, users with existing review suspensions, and users who have never performed a review).

Ability to easily cite problematic reviews

When an eligible user is loaded, their profile is shown (as well as the number of times that the user was suspended from reviews in the past, along with a link to show details on this), as well as a list of their 15 most recent reviews.

Step 2 of new Review Suspension UX for mods: Select reviews tasks from the most recent tasks performed by the user, for inclusion in the review suspension message
This table can be toggled using the drop down to show the last 15 tasks for the user in all review queues that are active on the site, as well as the last 15 audits (all audits and failed audits).

Table showing selected reviews for the user, which will be used in generating the review suspension message

The reviews shown in this table can be selected for inclusion in the review suspension message using the [Add task] button. Reviews not included in the table can be added directly using the text box above the table.

Customizable message templates

Modal giving a choice for which review suspension message template to use in generating the message for this user

We’re including options for 11 templates which were based in large part on templates that had already been in use by moderators, as well as feedback we received through other avenues. We also shared drafts of these messages with moderators on their Stack Overflow Team for a final round of revisions. The template messages can be previewed in the modal.

After clicking [Use this template], the moderator is shown the actual review suspension message that has been generated using the selected review tasks combined with the review suspension message templates. Below a set of review suspension message guidelines, the generated message shows up in a text editor, which is fully editable by the moderator.

A duration for the suspension must be selected at this point. These will default to the escalation rules that have been in place for automated audit-based bans since 2016 (start at 2 days, double the previous length whenever a new automatic suspension is started within 30 days of the end of a previous suspension, half the previous length if not).

Review suspension message shown to the user the /reviews section

Once the review suspension has started, the user will see the suspension message the next time that they access review queues. Messages are now rendered including line breaks and lists.

Current & Historical Suspensions

The screen showing current suspensions includes the same columns as before, using our new design palette, with more sorting and filtering options. The Reason column displays the message template that was used, which can be clicked to toggle display of the actual message:

New table showing current review suspensions. Columns: User, Started, Ends, Reasons, # Suspensions and button to Unsuspend the user. Reasons column shows the name of the message template that was used, and an expand/collapse function for the message that was sent.

New Help Center Articles

We have expanded the body of content available in the help center related to review queues. The goal is to provide reviewers with more education about how review queues work, and to have a canonical place to point them to when they get stuck or suspended. Queue-specific suspension messages will link directly to the Help Center article that is dedicated to that queue.

Metrics & Feedback

The metrics that we will be tracking to judge the overall success of these upgrades and new features will include:

  • Repeat reviewer suspensions and changes in duration: we hope that both these numbers will go down as improvements in product education and the quality of moderator guidance goes up.

  • Number of unique moderators creating review suspensions:

    • The old interface presented a barrier to entry for moderators to get involved with review suspensions and to create quality messages

    • New entry points to review suspensions make the feature easier to find and more efficient to access

  • Direct feedback from moderators about review suspensions

  • Visits to the suspension page, and conversion rate of visits to page through to actual suspensions

We are excited to be able to provide this new functionality to the moderator community, and are hoping that these features will contribute positively to the efforts of moderators who take an interest in reviews and content quality. A more detailed version of this post has already been shared with the moderator community and we are collecting feedback and bug reports directly from them. We are sharing it here in order to keep the overall Stack Exchange Community informed about changes to important site operations. We welcome your thoughts below.

  • 36
    kudo's for the excellent sock-puppet names! – Luuklag Nov 9 at 14:37
  • 43
    For testing we need 6 to 8 users to get review suspended. Comment below to become a volunteer. Any comment will do. – rene Nov 9 at 14:38
  • Feel free to use my account for that @rene.Depending on the site, whether it's "useful", because I only have review privs here and on QCSE. – MEE Nov 9 at 14:44
  • 6
    @MEE it will take another 6-8 years before rene is ready for that ;) – Luuklag Nov 9 at 14:54
  • 10
    Is there any way for me to tell how many review suspensions I've had myself? – DavidG Nov 9 at 16:27
  • no hats in there, no upvotes (just joking, have one. But thou shalt work on something else soon, right?) – BlueSoul Nov 9 at 16:41
  • 11
    I like how you called me "Albert Einstein". – Sonic the K-Day Hedgehog Nov 10 at 1:54
  • 11
    To add more detail to my comment above, I noticed that all the review tasks mentioned in the screenshots as belonging to user "Albert Einstein", all of those are reviews by me. The review IDs and timestamps all match my reviews here on this site. The user ID in the top one also matches mine. The screenshots seem to be from early March, as I remember having that much reputation and badges at that time (also confirmed in my /reputation log). I'm curious to know why you picked me... – Sonic the K-Day Hedgehog Nov 10 at 5:28
  • 17
    @SonictheK-DayHedgehog we do local dev work (where I took the screenshots) on snapshots from MSE. Last time I refreshed was around March. I needed to use an account for testing who had performed bunch of reviews. And I wanted to use someone who would appreciate being included, and who would also figure out that I was including them, despite the rename (otherwise, what's the fun?). – Yaakov Ellis Nov 10 at 8:17
  • 8
    That sounds quite interesting, and yes, I'd appreciate being included. I wonder how you chose the name, though... – Sonic the K-Day Hedgehog Nov 10 at 8:37
  • 6
    I guess you should feel honoured for being estimated smart @SonictheK-DayHedgehog – Luuklag Nov 10 at 8:40
  • Question: how are disputed bans handled? Like: user X gets auto-suspended, and appeals, and a moderator agrees to the appeal. Will those show up, too? Or some sort of "yes suspended, but appealed" information? – GhostCat Nov 11 at 12:32
  • 3
    @GhostCat if the moderator unsuspends the user before the suspension automatically expires, the user's review suspension history will show this (who did it and when). – Yaakov Ellis Nov 11 at 12:46
  • 1
    Now that it's live, where should I report bugs (that don't disclose information which isn't meant to be shared?) As an answer here, as a new question, or on the Moderator Team? – Glorfindel Nov 13 at 8:25
  • 1
    @glorfindel answer here or on mod team announcement for the next two weeks. After that as new post in either place. – Yaakov Ellis Nov 13 at 9:04
29

So my only suggestion would be to provide an interface similar to that of reputation that’s only accessible to the user.

So for transparency reasons, how difficult would it be, to provide a page for the user that indicates which audits they have failed? This page could also display when they were review suspended. This might help users realize that they have been suspended in the past.

It seems users are review suspended several times without actually realizing it.

| improve this answer | |
  • 1
    Why not make them a special case in the user review history. That would be an appropriate place if you ask me. You can then simply replace the action take by Failed audit. – Luuklag Nov 10 at 10:42
  • @Luuklag - However, it’s displayed, it’s important it’s only accessible to the user. A moderator can already access this information. – Ramhound Nov 10 at 10:59
  • Do you mean provide a page that indicates when they were suspended? Audit failure is not private. – TylerH Nov 11 at 19:06
  • 1
    @TylerH - My suggestion is that this specific feedback would be private. I understand audit failures are not actually private. It seems a never ending pattern of users being reviewed banned or failing audits only to ask about (2,3,6) month long review suspensions – Ramhound Nov 11 at 20:51
  • 2
    Good idea, we'll try to find a good place to put this so that a user has access to their own review suspension history – Yaakov Ellis Nov 15 at 8:40
15

Make it easier to tell when automatic suspensions were imposed for failing audits, and which failed audits were the cause of the suspension

The pages, as described here, do a great job at informing moderators of review audits a user failed as well as indicating when review suspensions were imposed for failing audits. But those two things are listed on two separate pages, which means it's harder to piece together the overall history of the user failing audits and how they led to a suspension.

Both of these items should be listed on the same page, so that it's easier for moderators to tell when and why a user was suspended for failing audits. Ideally, this would be in the review suspension history page: for automatic suspension entries, it should indicate the specific failed audits that led to that particular suspension (the three failed in the last 30 days, or the one failed within 30 days of coming off a previous suspension, per the current way those are imposed).

This way, if a user comes to meta and complains about being review-suspended, a moderator can point them to the specific review audit(s) that led to the suspension, which the user may have forgotten about.

Also, I've often seen cases where moderators are unaware of the rule that a single audit failure within 30 days of coming off a prior review suspension will lead to an immediate re-suspension. Most of the time, users come to meta sites complaining that they were suspended for a single failure, and moderators speak to them in a way that implies that they're lying ("suspensions aren't given for single failures", etc.). While that's technically true if you count the previous audit failures that led to the first suspension, from what I can tell, mods sometimes aren't aware of this rule.

And in the case where that single failed audit was actually a bad audit, it's hard for mods to go through and piece all the history and find that it was caused by that single failure.

| improve this answer | |
6

When looking at my own review (or one of my moderator colleagues), there's a link to 'suspend user from review'.

enter image description here

However, this is not possible, as the next page shows:

enter image description here

Just like you can't send a mod message to your own account, can the link be deactivated or even grayed out?

| improve this answer | |
  • 2
    I recall suspending myself from reviews while being a moderator (as a test), so this used to be possible. – fedorqui 'SO stop harming' Nov 13 at 10:03
  • 1
    Used to, but no longer. – Yaakov Ellis Nov 15 at 12:05
  • @YaakovEllis Why is review-suspending oneself no longer possible? This was used by mods in the past to test out bugs relating to review suspensions (in fact, the original impetus for revamping the review suspension system came from a feature request that was filed as a result of a self-suspension). – Sonic the K-Day Hedgehog Nov 16 at 6:50
  • 1
    @sonic I just didn't understand the use case for it. How about: no suspending a mod, except for yourself – Yaakov Ellis Nov 16 at 8:03
  • 2
    @YaakovEllis That makes sense. I don't see a need to suspend other mods, but giving the ability to suspend oneself makes it so that a mod doesn't need to make an alternate account to see what things look like for review-suspended users. – Sonic the K-Day Hedgehog Nov 16 at 8:18
  • 3
    Yeah, perhaps keeping the option to suspend oneself would be good to keep. A sockpuppet is easily made, but not one with 500 reputation ... – Glorfindel Nov 16 at 8:21
5

The filter for the type of review in Step 2 lists review types that might not exist on the site. For example, the site I moderate, Arqade, only has the Close Votes, Low Quality Posts, First Posts, Late Answers, Reopen Votes, and Suggested Edits review queues. As shown in the screenshot below, however, the filter in the new review suspension UI shows other review types, which of course don't return any review actions when selected.

Dropdown for the type of review

Could these be removed?

| improve this answer | |
  • 2
    Concretely, this means removing Triage and Help & Improvement on all non-SO sites, Site Self-Evaluation on all sites that weren't in beta prior to July 2015, and First Posts and Late Answers on all meta sites. – Sonic the K-Day Hedgehog Nov 13 at 19:36
  • 4
    This was the intended behavior, but accidentally forgot to use my list of enabled review queues for the site in this filter. Fixing. – Yaakov Ellis Nov 15 at 12:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .