Lacking confidence in my reviewing abilities, I usually avoid the review queue. But, I've been asking a decent amount of questions lately and haven't found many questions I could answer (so as to maintain karmic neutrality or whatever). So, I've been reviewing more lately. Wanting to improve my reviewer chops, I've been looking around for ways of getting feedback, but not finding much. Sure, sometimes the review system gives you a fake question, but those're pretty obviously good/bad.

Wanting to spur some good discussion, let's lay out some basics.

Here are some hypothetical examples:
(A) 85% of the questions I marked as "Looks OK" ended up being considered genuinely OK.
(B) 40% of the questions I flagged for "Requires Editing" ended up being closed for needing more detail.
(C) Typically when I flag a late answer to a question, it's subsequently unanimously marked as OK.

Note that, in each of these cases, I'm learning something specific and actionable about my reviewing approach. Also note that this information is nuanced: it's easy to see whether or not my "Looks OK" reviews match the consensus with a check or X UI element (as suggested here), and a percentage agreement with the consensus (as suggested here) would provide a more quantitative metric. But, neither would provide the more targeted feedback shown in hypotheticals (B) and (C). Equally importantly, the consensus isn't always right, so we shouldn't give this impression.

Not Outsmarting Ourselves
For several reasons (humans are imperfect, reviews are limited, etc.), we shouldn't push everyone to review exactly the same way. So, an important question in any such improvement is how to allow for some wiggle room? To me, this is the keystone to doing this right. Indicating that the consensus answer is always right/wrong is a problematic precedent. Perhaps the feedback system should compare how often my reviews deviated from the norm to how often others' reviews do so, i.e. normalizing the variance. What about a system that provides notifications when we're making the same mistakes? E.g. the next time I flag a late answer, a pop-up tells me that reviewers don't usually agree with me when I do this. In this way, a nudge is provided against my worst behaviors right as I'm making them.

As it stands, I've had a really hard time figuring out how to be a better reviewer, and it continues to be a deterrent to me reviewing items. This feels bad for the community, and it feels fixable. But, I'm far from a specialist in these matters, and don't have a good sense of what is doable. So, how could automated reviewer feedback be better conveyed? Do any of the ideas above hold water?

The zeroth-order solution would be this feature request or a slightly improved version in this one, though, as laid out in this question, I don't think it'd be hard to provide much more powerful feedback that also doesn't push people too hard toward reviewing the same way.

Reorganized, updated throughout to juxtapose more strongly to the two related questions (for which this question is currently marked a duplicate), and to emphasize how this question differs.

  • 1
    Just want to "maintain karmic neutrality or whatever", try this: What is a tag wiki? How do I write a good one? - Many sites need the work done and you earn +2 for each part, the first almost always shorter than any Q|A and the 2nd larger part often fairly easy.
    – Rob
    Dec 3 '19 at 1:06
  • @AntonMenshov Thanks, I actually didn't find that one while I was searching. I think my question here goes substantially deeper than just marking the consensus, and have updated the question accordingly. Marking the consensus can be both too broad and overly prescriptive.
    – TTT
    Dec 3 '19 at 1:41
  • 1
    Related thoughts I had a few years ago: meta.stackoverflow.com/questions/289871/…
    – PM 2Ring
    Dec 3 '19 at 2:17
  • @Rob this is effectively the same as the question that Anton marked, but the idea suffers from similar issues and limitations as what I'd mentioned in my edited question and comment back to him.
    – TTT
    Dec 3 '19 at 2:42
  • 1
    @Rob, simply put, neither of the two related questions would provide the feedback that I suggest in my three hypotheticals. I'll revise my question to try to emphasize the distinction.
    – TTT
    Dec 3 '19 at 3:51
  • TTT, reply to: To me, this is the keystone to doing this right. ... tells me that reviewers don't usually agree with me ...". - We don't want you to adjust your review to conform to other reviewers, we want every reviewer to review correctly - if you disagree with the other reviewers and you are correct that is excellent, we don't want to influence your reviews to agree with ones that were done incorrectly. The first How to Review has hints, search skills find a newer one. Make your Q an A for the UI.
    – Rob
    Dec 3 '19 at 4:59
  • 2
    I think the flags should be compared to post metrics, not to review metrics. So for posts voted "looks ok" it should consider posts that didn't receive downvotes for example.
    – Luuklag
    Dec 3 '19 at 9:31

This feels bad for the community, and it feels fixable.

Unfortunately, the second part is a misconception.

In order to fix this, SE Inc. would have to work closely with the community. Your question would be a good starting point for such activities. Like a question that I asked on MSO about one year ago: How to fix triage!.

I simply suggested to rework the labels on some buttons. And of course, outlined that "so many other things could be done". The most important one: we shouldn't be operating "in the blind". Nobody is really looking how these queues "perform", and what kind of "value" they generate for the community.

Also note: the triage/help concept on SO is completely broken. A lot of people are wasting their time with that, for almost no gain. That is not just an unpleasant experience that makes it harder for individuals to learn and enhance their review skills.

But, unfortunately, the answer I got last year still applies:

In a reasonable world, a reasonable suggestion like yours would be actioned promptly.

Stack Overflow the company is no longer a reasonable world; I'm not certain that they even have devs employed there anymore, except Marc Gravell. There's no other reasonable explanation for the fact that numerous useful changes to the Help and Improvement queue have been suggested since it was launched three-and-a-half years ago, yet almost none of them (see the answers in that question) have been implemented - even incredibly basic things like text changes, for crying out loud.

In other words: I really cherish your thoughts. But all of that is futile. Such proposals might lead to a bread crumb falling of the table for us occasionally, but beware: don't expect anything meaningful to happen. Sure, the community can have great discussions all day long. Won't change a thing.

With the-loop, SE Inc. clearly tells where we are heading. A world where feedback means two sentences "what you like" and then one "what you dislike". The idea that we start measuring the "inner mechanics" of the communities in the required detail, to then come up with visions and plans how to improve certain aspects and experiments and whatnot ... sorry, but if that is what we want, we are 8 to 10 years too late.

And just to be not completely negative: some weeks back, when announcing the changes to close reasons, Yaakov assured us that they are actively working on improving "such things". And I appreciate that. But I just don't see us working hand in hand here. Largely because SE Inc. suggests to put up very different feedback mechanism.

  • 3
    "SE has to work with the community" that's going to be your answer to just about every Meta question this period, isn't it?
    – Mast
    Dec 3 '19 at 13:05
  • 3
    @Mast When the question is "how to fix X", and "fixing X" requires that the community works with the company ... then explaining that "fixing X" has a low chance of happening, is well, the only thing left. As written: we can sit here all day long and discuss "we could this to fix X, or maybe that, or maybe something completely different". Also note that I dont just rant here. I am simply pointing out that there is a long long history of such requests. Another request wont help, unfortunately.
    – GhostCat
    Dec 3 '19 at 13:12
  • 2
    @GhostCatsaysReinstateMonica I disagree that requests won't help. Detailed requests explaining what is wrong with the current rules/implementations are useful for the community because they can be learned from and improved upon on other alternative platforms which might have similar issues. Dec 3 '19 at 15:05
  • 2
    @GhostCatsaysReinstateMonica, only time will tell, but I'd like this discussion to focus on the "what if" world in which change could happen. And in the worst case scenario where the SE network goes down in flames, these sorts of discussions could still help its successor.
    – TTT
    Dec 3 '19 at 15:19
  • @TTT Sure. Nonetheless, it is also true that esp. for SO, you can already find multiple such discussions. There is already plenty of input that SE Inc. could look into, and "people looking for inspiration for a new place", too. But also note: this stuff is all very specific. People building something new should rather come up with a real new vision, and pick up from here what might be useful. Instead of taking all the details and the "as is" of this place as building plan for a new site.
    – GhostCat
    Dec 3 '19 at 18:10
  • @TTT Seriously: people designing a new site ... should go a really different path. You get good designers and UX designers, and you first define your overall UX vision and whatnot. Focusing on stuff "because it is already there", and then trying to turn that thing into something much better ... by polishing a bell here and a whistle there is a very ineffective approach.
    – GhostCat
    Dec 3 '19 at 19:17
  • @GhostCatsaysReinstateMonica Apologies for the late reply. I don't know much about UI/UX. My specialties are more on the data/stats side, so my comment was with regard to leveraging the underlying data and what sorts of metrics might provide the most productive feedback to users. I DO think such ideas are useful across platforms.
    – TTT
    Dec 16 '19 at 15:59
  • @TTT Sure. One can always approach UI/UX from different angles. But what I have learned over the last years: it is better to start from the "end users perspective". Ask "what do you want to do/achieve", not with "what is possible". Spending a lot of time visualizing things ... that nobody needs to look at ... is most often less efficient. Hoping to uncover hidden secrets by chance is simply less likely to result into "big loops forward", compared to fully focusing on "limiting the search space based on user input first".
    – GhostCat
    Dec 16 '19 at 18:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .