76

On our announcement for our updates to review queue workflows, we got a ton of feedback on the Community user being the one posting comments for users. Here's a quick summary of some of the major points:

  • It's not clear to many users that Community is a bot that they can't reply to

  • Mods cannot delete comments from the Community user, and other users can't flag those comments, so they're just ... stuck there

  • Some reviewers don't mind having their name associated with their feedback

  • Community comments are showing up in the moderator dashboard

Relatedly, we got a good amount of feedback on the items in the Share Feedback action itself:

  • Some of the text in the feedback available is too code-specific, even in feedback that was meant to be more general

  • Users can't submit their own feedback through the Share Feedback option

After discussing these issues as a team, we came up with a solution that we believe resolves most of the issues with how we've implemented feedback sharing.

Here's our plan to resolve these issues:

  1. The Community user has received a "bot" label, similar to how I have a "staff" label and mods have a "mod" label. This now shows up next to the name of the Community user.

  2. Reviewers will have the option to submit their feedback as either the Community user or as themselves.

  3. Reviewers will be able to create their own custom feedback through the Share Feedback modal. This custom feedback can only be shared as the reviewer themself, not the Community user.

  4. Users who try to respond to the Community bot's posts will be informed, additionally, that they are replying to an unmonitored bot account (exact wording TBD).

  5. Mods will be able to delete Community bot comments, and other users will be able to flag the comments as "No longer needed."

  6. We're revisiting the text of some of the feedback strings to try to make it as general as possible.

We're specifically not addressing these issues right now:

  1. Creating blanket community-specific feedback options - this is a fairly large undertaking that we're not flat-out declining, we just don't have the resources to get to it right now.

  2. Creating a specific "Review" bot - like the item above, this is a big undertaking, and as most of the issues can be resolved by adding a "bot" label to Community, we're going with that solution instead.

  3. Reviewers can't undo feedback that they've sent in error - This will be able to be handled by flagging the comment as "no longer needed," so we're not going to create a specific way to undo feedback.

  4. The Community bot is only anonymous through obscurity (can be determined by looking at the post timeline) - We're willing to address this in the future if it seems like revenge downvoting is becoming a problem in this context, but we think that it's obscured enough right now that folks having their posts reviewed in these queues won't typically go searching for this information.

We wanted to create this as a separate meta post so that we can get any other feedback you might have on this solution. We're working on mockups for the designs now, and will be working on the fixes soon. If there are any cases or issues we haven't thought about regarding this solution, let us know! We'll be able to incorporate feedback into our fixes.

Please keep your feedback in this post to these specific issues - we are still monitoring the original release post and working on the other more standalone issues there. Note that we are addressing the Custodian badge issue from First Posts in a separate meta post.

And, as always, thank you all so much for your candid and thorough feedback. We genuinely want to make review queues as useful and easy to use as we possibly can, and we wouldn't be able to do that without your help.

11
  • 8
    I'm assuming (6) of the plan is going to include removing references to adding code on non-code sites? Aug 31 at 17:10
  • 18
    Respectfully, I would highly suggest that you review some actual posts (preferably on SO and on a non-code site). I looked at your profile at all the sites on which you earned any reputation and you don't have the Custodian badge on any of them, which means that you haven't reviewed a single post on those sites (same for Lisa Park, who only has two Custodian badges on SO). A lot of the problems I ran into with the new review queues were quickly apparent as soon as I started using them.
    – Null
    Aug 31 at 17:48
  • 3
    Will there ever be a way to mark normal user accounts as bot accounts? e.g. Natty posts automatic comments to which some people try to reply from time to time.
    – Dharman
    Aug 31 at 18:46
  • 3
  • 1
    The mod and staff badges appear based on the site (main vs meta vs MSO). Will the bot badge appear everywhere?
    – LShaver
    Aug 31 at 20:27
  • @Null I agree and want to emphasis the respectfully part, reviewing things well is hard and requires experience. It follows that any changes to the review process requires some of that experience, most preferably first hand.
    – Pureferret
    Sep 2 at 10:05
  • 2
    One very small sidenote - part of the friction here is that a lot of sites don't deal with code, and the default is sites that do code. I do realise this seems a wierd thing to suggest but - it might be useful to have 'non code' sites as a default
    – Journeyman Geek Mod
    Sep 2 at 15:34
  • 1
    I really feel like at least for the change you're making for #1, the community already let you know about this and at the time it was decided not to do it. I can appreciate different mentalities and different headspaces at the time, but to an outside observer (not me), it'd seem like y'all overlooked this feedback intentionally.
    – Makoto
    Sep 2 at 17:46
  • I think these are all great permanent solutions to the issues. You've never been able to retract a review before in a review queue, so I don't see that as a "bug that needs to be fixed as part of this implementation"... if anything, it is an independent feature-request. Thank you for the adjustments and for listening!
    – TylerH
    Sep 4 at 3:25
  • @badcoder The term "badge" is not incorrect, it is the usual expression for any permanently displayed title within Stack Exchange. Employees in the US are said to wear name badges, not name labels.
    – Mari-Lou A
    Sep 15 at 7:43
  • 2
    @rla4 please use the sandbox for tests like that
    – Luuklag
    Sep 16 at 4:46

13 Answers 13

39

First of all, thanks a lot for this detailed response to the Community's ... our feedback!

Users who try to respond to the Community bot's posts will be informed, additionally, that they are replying to an unmonitored bot account

Just a quick note on this: if they're using comment replies (@Community) that will be clear, but in my experience most new users don't know of that feature and just write a comment. Normally, this pings the author of the first comment as well, so this is not a problem. However, it could be very difficult to determine whether the user is trying to respond directly, or e.g. just adds additional information (which should often be an edit to the post).

6
  • 1
    That's a good point. Honestly, not totally sure how to resolve that, other than showing a warning to OP whenever they write any comment in the thread after Community has written a comment.
    – kristinalustig StaffMod
    Sep 1 at 13:05
  • 3
    @kristinalustig, something like the "[OP name] is a new contributor. Be nice, and check out our Code of Conduct." that appears above the comment box but uses wording of the ilk "Comments from Community [Badge] are automated and a reply won't be viewed by anyone" (but obvs. with much better wording!) might be a route to take?
    – Rob
    Sep 1 at 13:40
  • 3
    Yeah, that's what I was thinking! I think that'd work well.
    – kristinalustig StaffMod
    Sep 1 at 13:45
  • @Rob But you don't see that info for commenters, do you? Only posters?
    – TylerH
    Sep 4 at 3:26
  • @TylerH, I'm not sure what you're querying? I was suggesting that the new contributor 'banner' UI when commenting could be used (to show to the OP) when there are Community User attributed comments that they may be responding to
    – Rob
    Sep 4 at 10:37
  • @Rob It sounded like you were referring to a new contributor banner that shows up on new users' comments, which is what confused me (since such a thing doesn't exist). I understand now after your clarification what you meant.
    – TylerH
    Sep 7 at 13:29
20

Reviewers will have the option to submit their feedback as either the Community user or as themselves.

Sounds like a good plan. Could this option persist across reviews? It would be mildly annoying to have to check a box or select how you want the comment to be posted each time you do a review.

3
  • 5
    Personally I think if a user writes a custom comment they should give up anonymity. We don't want the community posting spam or rude comments.
    – pacmaninbw
    Aug 31 at 17:41
  • 9
    @pacmaninbw That aspect is already covered: "This custom feedback can only be shared as the reviewer themself, not the Community user."
    – Chindraba
    Aug 31 at 17:50
  • 3
    That's a good idea - I'll include that in the list of ideas/feedback we've gotten from this post and see if we can implement it.
    – kristinalustig StaffMod
    Sep 1 at 13:04
18

Please disallow audits from posts deleted via red-flags. It's not clear what you're supposed to do, for instance, when presented with blatant spam. "Share Feedback" is the "correct" choice for the audit, but it's not what we want reviewers to pick.

Spam Audit

4
  • Apologies if I posted this in the wrong question
    – Machavity
    Sep 1 at 20:24
  • 9
    Counterpoint: the actual issue here is that there is no correct response to spam in this queue, and that should be fixed instead of removing the audits.
    – Ryan M
    Sep 2 at 6:37
  • @RyanM correct option is to press the Skip.
    – bad_coder
    Sep 2 at 11:19
  • 10
    @bad_coder But that's not a reasonable experience: "Skip" is for "if you’re unsure about what to do." I am sure what to do about spam posts: flag them as spam. I should be able to complete the review after doing so.
    – Ryan M
    Sep 2 at 17:42
16

Thanks for bringing the suggestions about this important feature into a separate post. This should keep things more organized.

  1. Reviewers can't undo feedback that they've sent in error - This will be able to be handled by flagging the comment as "no longer needed," so we're not going to create a specific way to undo feedback.

Is a flag from the user who posted the comment going to be treated like NLN flags from other users or is it going to cause the comment to be deleted instantly as suggested here?

11
  • I’m not sure I understand. Are you thinking we won’t be able to delete comments we posted as ourselves?
    – ColleenV
    Aug 31 at 16:44
  • 2
    I think this is a very valid question; while it can probably be assumed that feedback submitted by "Reviewers ... as themselves" will be functionally the same as a normal comment by that user, I don't feel that the post made it 100% clear that this will be the case.
    – zcoop98
    Aug 31 at 16:50
  • 8
    @ColleenV I'm not talking about feedback that the users submit as themselves (I fully expect those to be posted as regular comments and be treated as such). I'm talking about comments posted under Community's name on the user's behalf. Since the user will not be able to delete/undo via a "Delete" button, then their NLN flag should instantly delete the comment. That was the fallback that I proposed in the linked answer. CC: zcoop98
    – 41686d6564
    Aug 31 at 16:57
  • 1
    Do they track “posted on behalf of” already? It seems to me like it should either be a Community bot comment or not. It would make it more difficult to anonymize that feedback in the future to keep a user associated with it. Also I wonder how that interacts with GDPR data requests, among other things.
    – ColleenV
    Aug 31 at 17:07
  • @ColleenV Yes, they do.
    – 41686d6564
    Aug 31 at 17:13
  • 1
    That’s information a human inferred from a timeline, not information a computer can look up in a database, like comment id 5785 is technically owned by user -1 but it was posted on behalf of user 10975434. A comment would have two “owners” and I don’t see how that’s not a huge change.
    – ColleenV
    Aug 31 at 17:16
  • Review comments made by Community are not the creation of the reviewer, and have the same delete-ability profile as a comment by anyone else. The only part of the process covered by GDPR should be that the review "reviewed" the post. If that data is removed by request then there remains no human-createable link between reviewer and the Community comment.
    – Chindraba
    Aug 31 at 17:57
  • @41686d6564 If a reviewer causes a Community comment, an NLN flag, or any other, should be processed the same as the same flag on a comment by any other user. If the reviewer wishes to retain "control" over the comments they also need to retain "ownership" of them.
    – Chindraba
    Aug 31 at 17:59
  • @Chindraba Two counterarguments: 1) It's not about control per se; it's more about cleaning up comments easily. If the reviewer is tracked and it's not too difficult to link back to them from the comment id, then why not allow the user to easily clean up (just like a single flag on a "thanks" comment directed at you deletes the comment instantly). 2) The comment is not 100% anonymous as indicated above. I can be tracked back to the reviewer. So, it only makes sense to give them some control over it.
    – 41686d6564
    Aug 31 at 18:05
  • 2
    From comments of those in the know, it seems that the user creating the comment is not tracked. Humans, using our reasoning skills can, sometimes, connect the comment to a reviewer. On a busy site even that might be less than perfect. 3 reviews closing at the same time, only one adding a comment, would leave the proper "association" to a guess.
    – Chindraba
    Aug 31 at 19:05
  • 4
    That's an interesting thought, and I'm not sure how immediately we'll handle those No Longer Needed flags. I'm going to put it on our list of things to consider, thanks for bringing it up.
    – kristinalustig StaffMod
    Sep 1 at 13:03
10

First of all a big thank you for the dedication and responding to feedback. It is much appreciated that you are willing to "go back to the drawing board" and change your original design choices based on our feedback. I'm sure that like me, all the community appreciates that.

A few things that already come to mind:

  • Are there going to be separate actions (buttons) for the two kinds of feedback - anonymous and regular? Or is it going to be under the same "Share Feedback" option? Personally I think two separate options will be best - one for anonymous feedback with the canned messages, and one for regular feedback with those canned messages and an additional free-form message. Mock-up example:

    enter image description here

  • Please consider removing the option to "Add a comment" if there's going to be an action to leave a free-form comment that finishes the review (especially as long as there is no "I'm Done" option). It might be confusing to have two ways to do the same (similar to how I reported about having both an edit link and an edit action in the Late Answers).

7
  • Out of curiosity, what would the benefit of having two separate options for pre-written vs. custom feedback? I imagine it being something where I'd look over the list of pre-written feedback options, then fill in a custom one at the bottom if none of them apply (similar to the custom close reasons UI).
    – Ryan M
    Sep 1 at 9:19
  • @RyanM I agree with you. I probably wasn't clear enough - I will edit. I meant two separate actions (buttons) - one for anonymous feedback, and one for regular feedback. Inside that regular feedback you would indeed have the canned options and another free-form option
    – Tomerikoo
    Sep 1 at 9:22
  • @RyanM See my gorgeous snipping and paint skilled mock-up xD
    – Tomerikoo
    Sep 1 at 9:42
  • Ahhhh, so it gives you the option to leave the canned feedback as yourself. Okay, that seems reasonable. It would eliminate the need to deal with either persisting or not persisting the option of whether to leave the feedback anonymously.
    – Ryan M
    Sep 1 at 9:45
  • @RyanM Exactly! This way it is still the same amount of clicks - "share feedback" -> choose option, but you are free to choose if it's anonymous or not every time
    – Tomerikoo
    Sep 1 at 10:34
  • 2
    I believe our plan right now is to have there be an option inside the modal to say whether the feedback you're leaving should be from the Community bot or attributed to you. I think we should avoid the word "anonymous" in the interface because it's only anonymous through obscurity.
    – kristinalustig StaffMod
    Sep 1 at 12:59
  • @kristinalustig Maybe the wording can be considered and changed, but I truly believe that having a check-box or whatever other way to indicate the poster's identity inside a single modal is going to be "annoying" to most users. It's only one more click, but it's for every review and some people review alot. Similar to the concern brought up here by TheLethalCarrot, which having two actions will solve. As I said in a previous comment, it will still be "Share feedback" -> choose option -> submit, and not "share feedback" -> choose option -> choose identity -> submit
    – Tomerikoo
    Sep 1 at 13:56
8

Regarding the feedback strings in First Answers: Many have complained about the code-oriented comments when the website is not about code. But in Stack Overflow, I find there is no appropriate comment for code-only answers.

According to the titles, I would say that "Answer needs supporting information" should be the most suitable, but it states:

Please add further details to expand on your answer, such as working code or documentation citations.

But there is working code. The problem is that there is only working code.

The other option, "Answer is unclear" is:

Please provide additional details in your answer. As it's currently written, it's hard to understand your solution.

Which is close but doesn't focus on the real problem.

Maybe you could add another option, something like:

Answer is code-only

Please provide some explanations to accompany the code you posted. Code-only answers are less likely to help future readers.

8

Add feedback strings for non-answers and link-only answers

A suggestion regarding this item:

We're revisiting the text of some of the feedback strings to try to make it as general as possible.

For first answers, it would be valuable to have some built-in feedback strings for various forms of NAA, with the goal of having new answerers be able to address the issues before the post gets deleted via low-quality answers review—I've seen plenty of link-only answers get fixed when it's pointed out, for instance.

The existing low-quality answers feedback options are a good reference for the types of issues we see frequently on Stack Overflow, but I imagine you probably don't want to add that many strings in addition to the ones that are more general feedback. I think they could potentially be condensed into two items: one general NAA and one for link-only answers. Possibly something like (with the link-only option lifted verbatim from the low-quality answers queue):

  • This does not provide an answer to the question. Answers should be an attempt to answer the question, rather than asking related questions, stating that you're having a similar problem, thanking other posters, or other commentary that does not answer the question.
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.

And as a postscript: These changes sound like a huge improvement; I certainly appreciate the effort to respond thoroughly to feedback and get this right.

2
  • 2
    I like the idea of providing an option for link-only answers. We'll talk about that as we're implementing these changes. I'm less certain about NAA option, because we're adding the post menu back in to the queues and that option exists in the post menu.
    – kristinalustig StaffMod
    Sep 1 at 13:01
  • 1
    @kristinalustig If by post menu you mean the option to flag, then as Ryan explained, it would still be nice to add a comment to help the OP fix the problems before it is further handled. They wouldn't know what to fix if we only flag without leaving some matching feedback
    – Tomerikoo
    Sep 1 at 15:37
8

This may be a bug or an intentional loophole, but by selecting Share Feedback it's possible to leave a comment even if the post is already deleted. Both as Community and as yourself. An example here on Ask Ubuntu:

enter image description here

4
  • 1
    Definitely looks like a bug. I'd get it if it were just for the Community bot because it's got mod privileges, but for a regular user... I retagged it. I hope you don't mind. Feel free to roll that back if you think it doesn't apply.
    – Ollie
    Oct 9 at 17:52
  • 1
    @Ollie the support was mostly for the Community part (to know whether that behaviour was intentional). I guess we'll know the answer in 6-8 weeks ...
    – Glorfindel Mod
    Oct 9 at 17:56
  • 1
    I doubt that the Community part's intentional; being able to leave comments, when you want and how you want, on a deleted post through a bot is still a bug, so...
    – Ollie
    Oct 9 at 18:05
  • @Ollie fair enough, you've convinced me :)
    – Glorfindel Mod
    Oct 9 at 18:15
7

I think you should immediately roll back all changes to the review queues until you can get all of these fixes in place.

Second, you're missing the greatest fix you need: to ensure that at all times the entire Stack Exchange network is in mind. I assume that dozens of people were involved in this update: from the individual developers, to their managers, to the Community Manager team, up to someone in the C-suite exec level. If that's the case, how did no one stop to think "maybe we shouldn't post a message about 'working code' to the humanities sites?" This update was so fatally flawed that it needs an internal investigation into how it could have been published.

Similarly, the fact that you made comments which mods cannot delete suggests that your testing regime is very deficient in including actual user review and moderator workflows. We mods were already frequently deleting the automated comments that came from the existing review queues. If you had talked to a single moderator they would have told you that undeletable comments is a terrible idea.

7

Some of these proposed changes are actively harmful.

The comments should not be made more general. The general nature of the comments is what makes them noisy and unhelpful. The whole idea of autogenerated comments should be thrown away, replaced with ways for users to leave more specific comments that point the poster in the right direction.

This would be resolved by saying that all comments must be left as the reviewer and only custom feedback will be accepted. If the user wants to leave anonymous feedback, they can do so with up votes, down votes, close votes, and/or delete votes. This would eliminate problems with users attempting to respond to Community, moderators not being able to clean up comments, or the poster being able to ping the commenter for more details and information.

I believe that requiring specific, custom feedback would eliminate the need for community-specific feedback options, undoing feedback (since comments can already be deleted), and remove the privacy-by-obsecurity aspects.

Please don't overcomplicate this. Just remove the Community comments.

5

This is not per se a response to your announcement of resolving some issues (for which, by the way, I'm grateful that at least the company is listening to meta feedback and can act quickly to roll back some harmful parts of new updates). I have a couple of issues, either bug reports or features that I don't understand, and a staff member suggested that I post them here.

  • Apparently the same post can now get enqueued twice for the First Answers review queue, even if the user doesn't have any other (even deleted) posts. Why?
  • Now that Late Answers has a Delete option, if you happen to get the same post in Late Answers and Low Quality Posts, once you've clicked Delete from one queue you can't do it again from the other, so you have to click Looks OK even if it's patently delete-worthy. This is part of a wider pattern of "Looks OK" being the only valid option even for non-OK posts in the new review system, making robo-reviewers a lot harder to spot from their activity.
4
  • I think you should post this here: meta.stackexchange.com/q/369013 Sep 7 at 18:12
  • 2
    @RandomPerson Originally I didn't want to post this on meta at all, because I don't have the time/bandwidth to bother hopping around finding the right post and so on. But a CM pushed me to post it here, so I took that as a sign that the company is listening to feedback posted here. Sep 7 at 18:42
  • Ok. Got it Rand al'Thor. Sep 7 at 18:43
  • 1
    Normally in the LA/LQP case I used to flag in LA, click I’m Done and then VTD in the LQP queue. I don’t believe this can happen anymore. Sep 7 at 20:34
5

Actions dialog on First Answers now has option to complete the review after flagging. But the name of that option is extremely confusing.

Previously that was named I am done which clearly indicated that you are done with all actions and want to complete the review.

New name is Other action which implied that you want to do some other action and it is not clear which one. It took me a while, to realize that is the proper option for completing the review.

Please rename this option back to "I am done".

You can also use some other, better name, but there was nothing wrong with "I am done" and renaming things just for the sake of renaming is bad UX.

If there are some other queues (I didn't go through all of them) with similar dialog, this feature request stands for them, too.

Actions Dialog

4

I definitely think this is a step in the right direction, but a relatively small step. It still doesn't resolve all the issues I have with feedback being from the Community user. I'll list each issue in my initial answer, and how this would address it:

1. It allows any user with 500 reputation to comment as official Stack Exchange

The Community user will receive a "bot" label, similar to how I have a "staff" label and mods have a "mod" label. This will show up next to the name.

Instead of thinking it's Stack Exchange, someone who didn't know about how the "Share Feedback" option works wouldn't know what to think. Not much better.

2. It's anonymous

Reviewers will have the option to submit their feedback as either the Community user or as themselves.

This is marginally better then forcing it to be anonymous, but users should still be responsible for their comments, regardless of whether they want to be or not.

3. Replies are broken

It's not as confusing if you add a bot label, but it still isn't very welcoming for a new user to have no way to get help from an actual human to resolve the issue with their post.

4. Users without knowledge of the new review queue workflow could think it's an auto-comment

This issue would definitely be resolved by adding a "bot" label, but a new issue will arise: users will think the post was reviewed by a bot and it automatically left a comment based on an algorithm.

5. You can upvote your own comment.

I didn't read anything in the post addressing this.


I'll ask again: what is the reason for having comments be from Community?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .