1

Most of the time I'm active on StackOverflow is scrolling through the Review Queues I have access to. These review queues are important to SO's operation, and going through them is my menial way of giving back to a service and community that has benefitted me immensely over the years.

But the system seems fundamentally flawed to me.

StackOverflow does an excellent job of crowd-sourcing QA effort, except on the QA effort itself. This answer outlines the issue very clearly:

We incentivize asking questions. We incentivize answering them. We incentivize making questions better by editing them (for people with lower rep). However, all of these incentive processes have review.

You only get rep from questions when people upvote them. You only get rep from answers when people upvote them. You only get rep from edits when people approve them. In every case, some human being has to look at what you did and say, "Good".

However, despite outlining the problem excellently, I believe it has reached the incorrect conclusion; the problem is not that reviews are incentivized, but rather that reviews are not reviewed.

Of course, there should not be review queues of review queue entries. But consider the concern mentioned in this post:

I went to a Late Answers review queue that had four items in it and started reviewing the first one by editing it into shape rather heavily.

By the time I was done, someone else had reviewed the same post simply by upvoting it, so I only got a "Next" button, not an "I'm done" button. My edit still counted as an edit; but no longer as a review.

Worse still, the other three items were gone from the review queue as well. Curious, I checked the review tab in the profile of the user who had snatched the first review out from under me. Sure enough, he also "reviewed" the other three items, all within a minute, without even fixing obvious typos, simply by upvoting. I had to go through his review list and review every single post again.

As soon as you add any sort of incentive to a task (even if it's just a silver medal), people are going to try and find the easiest possible method of completing that task. Without any way of performing quality assurance on reviews, the situation outlined here will keep happening.

And as important as this concept is from the perspective of pruning out bad reviews, it is equally important to incentivize good reviews. In the situation outlined above, the writer had clearly put significant effort into improve a late answer, but clearly take issue with the incentive disparity for doing that task properly vs. shoddily. I would wager that a large majority of users ignore the review queue because it's not a good means for rep gains (anecdotally, my rep gains have certainly slowed since I started browsing it).

You want to attract high-quality feedback in the review queues, and the current system (including the automated auditing) does not.

As a discussion point:

  • Is there a good reason not to incentivize high-quality review queue feedback?

As a feature request:

  • Allow visibility of reviews performed, and voting (both up and down) on them. StackOverflow has proven that crowd-sourced QA works; let it work for performing QA on the QA.
  • Note that you don't need to upvote every post to pass a review. The fastest way to just go through reviews without reading them is to just click "no action needed". In fact that button was created specifically because people used to just upvote every post without reading, and we wanted to make sure that the easiest way to abusively finish a review wouldn't have as harmful of negative effects. – Servy Jan 26 '18 at 14:56
  • While fair, I'd argue that preventing abusive reviews from being harmful is not as good as being able to be punitive against abusive reviews (and, likewise, incentivize quality reviews). – Ironcache Jan 26 '18 at 15:01
2

You're right that we could be doing a bit better about rewarding the behavior that we want, but it gets to become a bit of a double-edged sword.

Let's say that we overhauled the review stats to basically show just how effective you were as a reviewer. How many times did you say a post should be closed that actually got closed? How many times did something you worked on in the Help & Improvement queue actually go on to do better than it would have otherwise? How much low-quality, essentially unsalvageable junk was cleaned off the site before it got in people's faces?

Knowing the outcome of how we'll you're performing there is way more valuable than a badge for completing an arbitrary number of actions unless you really stink at doing it. That's why we have the audits, temporary breaks, etc.

I think there's a way to show genuinely good effectiveness for 2/3 of the folks doing it without making the last third look and feel really bad in the process, but I haven't quite put my finger on what that is. And, well, it could be that being brutally honest with 1/3 of the people using the system by saying you really stink at this with graphs on how effective they've actually been is the best thing we can do, but it's certainly not the nicest, which makes it an interesting problem.

I'm open to suggestions, I'd love to do something that gives effective reviewers something pretty shiny to admire and show off, but the cost to the bottom third that really stink at doing it would be pretty harsh. To be clear, they aren't really harmful to the system, they're just not that good at doing it.

You don't have a bad idea, it's just one of those things that starts to get really icky once you get into really any kind of implementation.

  • How often someone's actions were agreed on by others isn't a particularly good indicator of how good of a reviewer they are, what matters is whether that action is correct. Trying to tell people that doing the same thing that other people would have done is "correct" isn't beneficial, particularly when there are a lot of bad reviewers out there. – Servy Jan 26 '18 at 15:19
  • @Servy I feel that it can be helpful when you measure a positive. How many posts were left open and went on to get upvoted answers? How many questions edited in the helper queue went on to get some upvotes, as well as answers and views? When you look at the net positive stuff it tends to speak to a much larger consensus than what becomes possible with a few bad reviewers agreeing on the same faulty premise. But you have to look more at what happened to things months later. – Tim Post Jan 26 '18 at 15:24
  • Thank you for the detailed answer. Does StackOverflow not already adopt the model of telling the bottom 1/3rd (as a somewhat arbitrary number) that they stink though? If you give an question/answer deemed as low-quality by the general public, you get down voted. Likewise, up voted for high-quality. So my question on this front would be is the system ineffective for Q&A? And, if not, would it be less effective for review feedback than Q&A feedback? – Ironcache Jan 26 '18 at 15:26
  • @TimPost The problem is that the actions people would take to make those "positive" stats go up are not necessarily going to be the correct actions to take. If you tell people how many of the posts they left open went on to get upvoted answers, they're likely to try to leave everything open, because that will result in more of them having upvoted answers. If you look at how many upvotes a post they edited got, they're likely to try to find the best posts and make trivial edits, rather than trying to fix posts with real problems, because that will inherently attract more upvotes, etc. – Servy Jan 26 '18 at 15:30
  • I would just expose all efforts the individual took when presented with the review, and let the community decide if their actions (not the outcome) were right or wrong (or an action that they have no strong opinion one way or another on). IE: User marked a blatently low-quality first post as "Looks OK"? Down vote. User submitted a comment telling the user how to better construct their question (present and future), an edit to fix grammatical errors due to ESL, and an answer that helped the user? Up vote. – Ironcache Jan 26 '18 at 15:36
  • @Servy That's exactly where we got stuck when we were thinking about it while also looking at the last profile page redesign. Presumably, a lack of progress for closing on top of great progress on reopening could paint a more realistic picture (similar to how credit scores work, you're rewarded for paying on time, but also penalized for using more than 60% of your credit, but not the best metaphor). It got, for lack of a better word, hard - which is why we didn't do it (and what I was explaining in the post). – Tim Post Jan 26 '18 at 16:02
1

Is there a good reason not to incentivize high-quality review queue feedback?

Sure, it's basically impossible to do. How do you plan to do it?

Allow visibility of reviews performed, and voting (both up and down) on them. StackOverflow has proven that crowd-sourced QA works; let it work for performing QA on the QA.

This already exists. There's a history for reviews for users with enough rep, and every users' reviews are shown on their profile.

  • Can you up vote and down vote that history? Are there rep changes associated with that history? Is there going to be any likely repercussions to me putting in the minimal effort on a review? Is there going to be any likely incentivizations for me putting in a lot of effort on a review? Does this actually promote high-quality feedback? – Ironcache Jan 26 '18 at 14:59
  • @Ironcache If you just want to know the basics of how reviews work you probably want to start by reading through the FAQs, actually doing a bit of reviewing yourself, searching through past questions on the topic, etc. – Servy Jan 26 '18 at 15:02
  • 1
    @Ironcache You can't vote on these. What you can do, is look for a pattern of bad reviews - at least for the Suggested Edits queue, and under some circumstances for the Low Quality queue. If you see a pattern of bad reviews, you can raise a custom moderator flag on one of the bad reviewer's posts and explain the problem. The flag should include the examples of recent bad reviewing. (As a rule of thumb, include 3 bad reviews in the past 2 days.) Needless to say, this is an arduous process. – S.L. Barth Jan 26 '18 at 15:03
  • @Servy I do use the review queues (which was mentioned in the post), and I have searched through past questions / wiki entries on the topic (also linked in the post). I simply have not come across a feature akin to what I've requested in the post, and (based on S. L. Barth's comment), it seems to be that it's because the feature I requested in the post does not exist. – Ironcache Jan 26 '18 at 15:09
  • @Ironcache If you know the answers to those questions that you just asked, then why ask them? Why say that you're asking them sincerely when you already know the answer (meaning you aren't asking them sincerely). – Servy Jan 26 '18 at 15:12
  • I never claimed to know the answer. I've simply not come across the answer. It's possible that what I requested does exist in some fashion. But what is mentioned in this answer is not the feature requested. – Ironcache Jan 26 '18 at 15:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .