Resolving Community user and "share feedback" issues in review queues

Question

On our announcement for our updates to review queue workflows, we got a ton of feedback on the Community user being the one posting comments for users. Here's a quick summary of some of the major points:

• It's not clear to many users that Community is a bot that they can't reply to

• Mods cannot delete comments from the Community user, and other users can't flag those comments, so they're just ... stuck there

• Some reviewers don't mind having their name associated with their feedback

• Community comments are showing up in the moderator dashboard

Relatedly, we got a good amount of feedback on the items in the Share Feedback action itself:

• Some of the text in the feedback available is too code-specific, even in feedback that was meant to be more general

• Users can't submit their own feedback through the Share Feedback option

After discussing these issues as a team, we came up with a solution that we believe resolves most of the issues with how we've implemented feedback sharing.

Here's our plan to resolve these issues:

1. The Community user has received a "bot" label, similar to how I have a "staff" label and mods have a "mod" label. This now shows up next to the name of the Community user.

2. Reviewers will have the option to submit their feedback as either the Community user or as themselves.

3. Reviewers will be able to create their own custom feedback through the Share Feedback modal. This custom feedback can only be shared as the reviewer themself, not the Community user.

4. Users who try to respond to the Community bot's posts will be informed, additionally, that they are replying to an unmonitored bot account (exact wording TBD).

5. Mods will be able to delete Community bot comments, and other users will be able to flag the comments as "No longer needed."

6. We're revisiting the text of some of the feedback strings to try to make it as general as possible.

We're specifically not addressing these issues right now:

1. Creating blanket community-specific feedback options - this is a fairly large undertaking that we're not flat-out declining, we just don't have the resources to get to it right now.

2. Creating a specific "Review" bot - like the item above, this is a big undertaking, and as most of the issues can be resolved by adding a "bot" label to Community, we're going with that solution instead.

3. Reviewers can't undo feedback that they've sent in error - This will be able to be handled by flagging the comment as "no longer needed," so we're not going to create a specific way to undo feedback.

4. The Community bot is only anonymous through obscurity (can be determined by looking at the post timeline) - We're willing to address this in the future if it seems like revenge downvoting is becoming a problem in this context, but we think that it's obscured enough right now that folks having their posts reviewed in these queues won't typically go searching for this information.

We wanted to create this as a separate meta post so that we can get any other feedback you might have on this solution. We're working on mockups for the designs now, and will be working on the fixes soon. If there are any cases or issues we haven't thought about regarding this solution, let us know! We'll be able to incorporate feedback into our fixes.

Please keep your feedback in this post to these specific issues - we are still monitoring the original release post and working on the other more standalone issues there. Note that we are addressing the Custodian badge issue from First Posts in a separate meta post.

And, as always, thank you all so much for your candid and thorough feedback. We genuinely want to make review queues as useful and easy to use as we possibly can, and we wouldn't be able to do that without your help.

Answer 1

First of all, thanks a lot for this detailed response to the Community's ... our feedback!

Users who try to respond to the Community bot's posts will be informed, additionally, that they are replying to an unmonitored bot account

Just a quick note on this: if they're using comment replies (@Community) that will be clear, but in my experience most new users don't know of that feature and just write a comment. Normally, this pings the author of the first comment as well, so this is not a problem. However, it could be very difficult to determine whether the user is trying to respond directly, or e.g. just adds additional information (which should often be an edit to the post).

Answer 2

Reviewers will have the option to submit their feedback as either the Community user or as themselves.

Sounds like a good plan. Could this option persist across reviews? It would be mildly annoying to have to check a box or select how you want the comment to be posted each time you do a review.

Answer 3

Please disallow audits from posts deleted via red-flags. It's not clear what you're supposed to do, for instance, when presented with blatant spam. "Share Feedback" is the "correct" choice for the audit, but it's not what we want reviewers to pick.

Answer 4

Thanks for bringing the suggestions about this important feature into a separate post. This should keep things more organized.

1. Reviewers can't undo feedback that they've sent in error - This will be able to be handled by flagging the comment as "no longer needed," so we're not going to create a specific way to undo feedback.

Is a flag from the user who posted the comment going to be treated like NLN flags from other users or is it going to cause the comment to be deleted instantly as suggested here?

Answer 5

First of all a big thank you for the dedication and responding to feedback. It is much appreciated that you are willing to "go back to the drawing board" and change your original design choices based on our feedback. I'm sure that like me, all the community appreciates that.

A few things that already come to mind:

• Are there going to be separate actions (buttons) for the two kinds of feedback - anonymous and regular? Or is it going to be under the same "Share Feedback" option? Personally I think two separate options will be best - one for anonymous feedback with the canned messages, and one for regular feedback with those canned messages and an additional free-form message. Mock-up example:

  

• Please consider removing the option to "Add a comment" if there's going to be an action to leave a free-form comment that finishes the review (especially as long as there is no "I'm Done" option). It might be confusing to have two ways to do the same (similar to how I reported about having both an edit link and an edit action in the Late Answers).

Answer 6

Regarding the feedback strings in First Answers: Many have complained about the code-oriented comments when the website is not about code. But in Stack Overflow, I find there is no appropriate comment for code-only answers.

According to the titles, I would say that "Answer needs supporting information" should be the most suitable, but it states:

Please add further details to expand on your answer, such as working code or documentation citations.

But there is working code. The problem is that there is only working code.

The other option, "Answer is unclear" is:

Please provide additional details in your answer. As it's currently written, it's hard to understand your solution.

Which is close but doesn't focus on the real problem.

Maybe you could add another option, something like:

Answer is code-only

Please provide some explanations to accompany the code you posted. Code-only answers are less likely to help future readers.

Answer 7

Some of these proposed changes are actively harmful.

The comments should not be made more general. The general nature of the comments is what makes them noisy and unhelpful. The whole idea of autogenerated comments should be thrown away, replaced with ways for users to leave more specific comments that point the poster in the right direction.

This would be resolved by saying that all comments must be left as the reviewer and only custom feedback will be accepted. If the user wants to leave anonymous feedback, they can do so with up votes, down votes, close votes, and/or delete votes. This would eliminate problems with users attempting to respond to Community, moderators not being able to clean up comments, or the poster being able to ping the commenter for more details and information.

I believe that requiring specific, custom feedback would eliminate the need for community-specific feedback options, undoing feedback (since comments can already be deleted), and remove the privacy-by-obsecurity aspects.

Please don't overcomplicate this. Just remove the Community comments.

Answer 8

Add feedback strings for non-answers and link-only answers

A suggestion regarding this item:

We're revisiting the text of some of the feedback strings to try to make it as general as possible.

For first answers, it would be valuable to have some built-in feedback strings for various forms of NAA, with the goal of having new answerers be able to address the issues before the post gets deleted via low-quality answers review—I've seen plenty of link-only answers get fixed when it's pointed out, for instance.

The existing low-quality answers feedback options are a good reference for the types of issues we see frequently on Stack Overflow, but I imagine you probably don't want to add that many strings in addition to the ones that are more general feedback. I think they could potentially be condensed into two items: one general NAA and one for link-only answers. Possibly something like (with the link-only option lifted verbatim from the low-quality answers queue):

• This does not provide an answer to the question. Answers should be an attempt to answer the question, rather than asking related questions, stating that you're having a similar problem, thanking other posters, or other commentary that does not answer the question.
• While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.

---

And as a postscript: These changes sound like a huge improvement; I certainly appreciate the effort to respond thoroughly to feedback and get this right.

Answer 9

bug

This may be a bug or an intentional loophole, but by selecting Share Feedback it's possible to leave a comment even if the post is already deleted. Both as Community and as yourself. An example here on Ask Ubuntu:

Answer 10

I think you should immediately roll back all changes to the review queues until you can get all of these fixes in place.

Second, you're missing the greatest fix you need: to ensure that at all times the entire Stack Exchange network is in mind. I assume that dozens of people were involved in this update: from the individual developers, to their managers, to the Community Manager team, up to someone in the C-suite exec level. If that's the case, how did no one stop to think "maybe we shouldn't post a message about 'working code' to the humanities sites?" This update was so fatally flawed that it needs an internal investigation into how it could have been published.

Similarly, the fact that you made comments which mods cannot delete suggests that your testing regime is very deficient in including actual user review and moderator workflows. We mods were already frequently deleting the automated comments that came from the existing review queues. If you had talked to a single moderator they would have told you that undeletable comments is a terrible idea.

Answer 11

This is not per se a response to your announcement of resolving some issues (for which, by the way, I'm grateful that at least the company is listening to meta feedback and can act quickly to roll back some harmful parts of new updates). I have a couple of issues, either bug reports or features that I don't understand, and a staff member suggested that I post them here.

• Apparently the same post can now get enqueued twice for the First Answers review queue, even if the user doesn't have any other (even deleted) posts. Why?
• Now that Late Answers has a Delete option, if you happen to get the same post in Late Answers and Low Quality Posts, once you've clicked Delete from one queue you can't do it again from the other, so you have to click Looks OK even if it's patently delete-worthy. This is part of a wider pattern of "Looks OK" being the only valid option even for non-OK posts in the new review system, making robo-reviewers a lot harder to spot from their activity.

Answer 12

feature-request

Actions dialog on First Answers now has option to complete the review after flagging. But the name of that option is extremely confusing.

Previously that was named I am done which clearly indicated that you are done with all actions and want to complete the review.

New name is Other action which implied that you want to do some other action and it is not clear which one. It took me a while, to realize that is the proper option for completing the review.

Please rename this option back to "I am done".

You can also use some other, better name, but there was nothing wrong with "I am done" and renaming things just for the sake of renaming is bad UX.

If there are some other queues (I didn't go through all of them) with similar dialog, this feature request stands for them, too.

Answer 13

I definitely think this is a step in the right direction, but a relatively small step. It still doesn't resolve all the issues I have with feedback being from the Community user. I'll list each issue in my initial answer, and how this would address it:

1. It allows any user with 500 reputation to comment as official Stack Exchange

The Community user will receive a "bot" label, similar to how I have a "staff" label and mods have a "mod" label. This will show up next to the name.

Instead of thinking it's Stack Exchange, someone who didn't know about how the "Share Feedback" option works wouldn't know what to think. Not much better.

2. It's anonymous

Reviewers will have the option to submit their feedback as either the Community user or as themselves.

This is marginally better then forcing it to be anonymous, but users should still be responsible for their comments, regardless of whether they want to be or not.

3. Replies are broken

It's not as confusing if you add a bot label, but it still isn't very welcoming for a new user to have no way to get help from an actual human to resolve the issue with their post.

4. Users without knowledge of the new review queue workflow could think it's an auto-comment

This issue would definitely be resolved by adding a "bot" label, but a new issue will arise: users will think the post was reviewed by a bot and it automatically left a comment based on an algorithm.

5. You can upvote your own comment.

I didn't read anything in the post addressing this.

---

I'll ask again: what is the reason for having comments be from Community?