20

What I've noticed is that on many sites reviews fall by the wayside. They either don't get done, or there's some other major problem with how they're being done (e.g., robo-reviewing). This is most obvious with suggested edits for the low rep crowd (who will be blocked from suggesting new edits when the queue is full), such as recently on Webapps, where moderators didn't see an accurate count of pending edits. I'm also seeing other large queues on other sites, such as close votes, which can balloon in size and bleed expired close votes and flags. Sometimes, like on Stack Overflow in particular, these problems are known but have no long-term solution. However, I'm seeing a real pattern where these problems with review are happening on other sites, nobody says anything (if anyone noticed), and even moderators are unaware, though there are often solutions out there (the easiest one in the short term being for some people to do reviews).

Since I doubt any warnings would be built into the system, this may have to be done by SEDE (or maybe the API). The question then becomes who will be responsible for running the queries. Ideally, this would be something that CMs monitor and bring to the attention of site moderators (including helping them to get longer term solutions where applicable), but failing having that in place, would anyone from the community take on the task (either of making queries or being the one to report problems to moderators)?

Below I have some ideas for health indicators. Some of these are indisputably problems, others only potential problems. Feel free to let me know if I missed anything that could be useful to monitor.

All queues

  • Old or aging-out reviews. Exactly how this works varies by queue, with some aging out at 14 days and suggested edits not aging at all
  • Review "hoarding": Most reviews done by a single user. Note that this should exclude low-rep users reviewing edits on their own posts.

Close/Reopen Votes

  • A pattern of "rogue" voting (e.g., only 1 Close versus 3 Leave Open)

Suggested Edits

  • Many edits in the queue (RIP suggested edit count in the top bar)
  • 4 or 5 pending edits from a single user

Low quality posts/answers

  • A pattern of "rogue" reviews
  • There are already autoflags for "disputed" reviews (which are similar), not sure if we would need more monitoring for that

First questions, First answers, and Late answers

  • Many contradictory reviews (e.g., Looks OK on a post that gets deleted)

Triage

  • I have no idea how this works anymore, but maybe I'll learn again and put suggestions here in the future.
9
  • 5
    In the low quality queue on non-Triage sites, you can get one person reviewing Looks OK outweighing 25 users reviewing to recommend closure. Commented Dec 27, 2023 at 3:44
  • 11
    In some ways this is a reflection of the more global issue of moderation participation and tooling. The idea of community moderation should extend to review queues. There should be enough users actively involved in reviewing posts and also even more importantly reviewing other reviewers. But with the queues tucked so far out of the way, high rep requirements (especially for review history), notice blindness on the perpetually red circle, little to no incentives for reviewing well, and no clear way to flag (or otherwise dispute) a review or review task it's just not working. Commented Dec 27, 2023 at 6:12
  • 2
    Who will review the review reviewers? Commented Dec 27, 2023 at 9:48
  • 7
    There needs to be more feedback for reviewers. How many questions did I vote to close that stayed open and vice-versa? How many questions did I vote to close that got reopened? How many of my comments were flagged as rude? I think review queues suffer from participation because it feels like your time is going into a black hole. On someone's activity page, you can see their voting stats, but not their review stats. I don't think the best solution is top-down. I think we need to encourage more participation to solve the issues, but data collection is a good first step.
    – ColleenV
    Commented Dec 27, 2023 at 13:26
  • @HenryEcker While I'm sure both of us and any number of other users here could come up with good tooling suggestions, it's not useful if developer time won't be spent on it. That's why I'm only focusing on work that the community can do by itself, or maybe with CMs.
    – Laurel
    Commented Dec 27, 2023 at 18:09
  • @ColleenV While the suggestions I have here are based on the health of the community overall, I would also be interested in queries to assess the performance of specific reviewers (both for self-reflection and moderation purposes). Unfortunately, there's not a good way to fix that systematically; like I said we can't rely on developer time being allotted to change this.
    – Laurel
    Commented Dec 27, 2023 at 18:12
  • 3
    @Laurel I think it would be counter-productive to use the stats that help reviewers align themselves with the community for moderation purposes, but I do think moderators need much better tools to investigate the interactions between specific accounts. I have been thinking of writing up my thoughts on how SE is focused on the wrong things. They think their purpose is managing information, when all the real value is in the communities. One human comment-discussion is worth 1000 AI-generated search summaries when it comes to getting new folks to engage with the network.
    – ColleenV
    Commented Dec 27, 2023 at 19:05
  • 3
    Every time I start writing though, I see that quote about not teaching pigs to sing that I keep on my desk and think better of it. I have more productive uses of my time, which makes me a bit sad if I'm honest.
    – ColleenV
    Commented Dec 27, 2023 at 19:07
  • 1
    WebApps has a host of moderation problems. This is probably the least of them, tbh.
    – Richard
    Commented Jan 3 at 0:50

0

You must log in to answer this question.