Taking action to improve First Posts and Late Answers reviewing - Upvote is too easy

Question

I found good questions about this subject (like that one) but they weren't quite narrowing to what I wanted to ask. So I wrote my own.

The problem

I have been reviewing for a few months now (~725 reviews) in those two sections (Late Answers and First Posts) and I have seen so many upvote on question that doesn't really deserved it. (e.g. Here and here)

In those case, I was almost angry about the reviewers that upvote them, feeling that they didn't took anytime to understand what's the question, which answers are already there, is the question already solved, how much the post they are reviewing is adding value to the thread, etc.

It happens often and during I'm reviewing posts. I'm still reading through and suddenly 1 then 2 and even 3 upvotes in a row.

I think some user don't put enough time in a review, maybe the Upvote option is so easy or at some point they don't really bother to read it entirely and they just want to get their daily cap.

In depth analysis of the situation

Maybe they don't understand the impact of a bad review on the community on the long term. The thing is those First Posts and Late Answers are the best way to educate new users to SO (or any other) and explain them how it works.

If people just up-vote them quickly without putting enough time evaluating which action would be best, we will lose the opportunity to show to those new users how it works.

When they got upvote 2-3 times for a bad answer, for instance, they will think it's the way to write them. They will come back later posting similar bad answer elsewhere. However, that time the reviewing system won't catch it!

I know there is a limit on the number of posts you can review per day. There is also if you reached your daily vote limit. Both helps, but I don't find it enough to prevent those kind of bad reviews.

Some suggestions

I thought about few things, but I'd like to know what other people think about this.

Limit the number of Upvote in the context of a review

For example, if you are allowed to review 20 Late Answer and 20 First Post a day, you have a limit of 10 Upvotes. Think about it, how many times half the reviewing post will really deserve an upvote? You're are lucky if you got 3 in a row IMO.

Find a way informs users how to perform a better review

Maybe there is not enough emphasis about how to make a good review? The first time they access the Review system, we could show them a page to teach them what is a good and a bad review.

Increase the reputation needed to make those review to 500

It honestly took me a little while to understand a lot of aspects of the system : what is the mentality when reviewing, what should be done. I often had questions myself and I went on Meta reading threads to know what should I do in some cases. For instance, when the answer is very similar to another one, which flag I should use, etc.

I don't think a lot of people actually do that. That's why I think there should be a lower limit of 500 rep, so users have enough experience to make proper review. Because even though they are the first type of review users can do, they are important IMO.

EDIT :

I don't want to target anyone, but this enforce my point. I have seen users (member since 2-3 months) with more than 2000 reviews done in total and they have literately ~1 500 vote cast where only ~10 are voted down. I highly doubt that so many reviewed posts were good enough to be up-voted. The saddest part of this, is that even with a limit of 1k rep (as suggested by Kevin), some of them would not be affected by that limit.

Answer 1

update

Equally easy but hopefully less damaging review option has been introduced.

"No Action Needed" button has been added in First Posts and Late Answers review queues. It increases review count but does not require reviewer to vote.

Answer 2

I quietly suspect that the root of the problem is that a large and well-understood part of the purpose of Review is to check for a few fairly objective Bad Things - including, but not limited to, spam, non-questions, and non-answers - but if none of those issues are present, there's no way of simply saying "Nothing is horribly wrong here" except an upvote.

I am a quite inexperienced programmer, and very frequently in review I'm faced with a question or answer that I can see is written in clear and correct English, is not spam, and is not patently idiotic... but it is in a field that I know nothing whatsoever about, so I can't reliably discern much more than that. I'm often tempted to upvote these just as a way of signalling "Everything looks okay here", but instead usually click 'Skip' instead - often after having invested minutes of my time researching around the subject matter, but without learning enough to be confident that an upvote is appropriate. This is really off-putting. It leaves me knowing that the minutes of time I've spent reviewing that question will serve no purpose for anyone.

But then a doubt comes into my mind: maybe I should just upvote? I vaguely understand the question; it appears reasonable; from his writing style, the OP seems to be sensible and know what they're talking about; I'd be willing to bet that an upvote is appropriate here. Why let my careful reading and thought go to waste, especially when I notice that three other people have smacked the upvote button while I've been pondering the question? Basically, the whole system at present encourages me to click the upvote button when I'm in doubt.

This experience makes me suspect that many of the upvotes on first posts that people on meta tend to attribute to bad people 'gaming' the review system for badges are really just down to people honestly upvoting questions they don't understand, but which appear, superficially, to be reasonable questions.

I wonder if the best mitigation to the problem is not to make upvoting more difficult, but to provide an equally easy alternative that doesn't send out the potentially misleading signals that an upvote does. Why not simply have an Everything looks fine here checkbox that you could check to let you click the I'm Done button without having to upvote the post you were reviewing? Honest reviewers would be able to indicate that a post appears harmless without having to (perhaps inappropriately) upvote it, the lazy and unscrupulous badge farmers could still lazily and unscrupulously badge farm but in a quieter way that would cause less collateral damage for the site, and everyone would be mildly happier than they are now.

Answer 3

If reviewers are upvoting questions or answers that have serious, visible problems - ones that get closed or deleted (for any reason other than being duplicates or deletion by the original poster) - then they're not doing their job and we should punish them for it.

Misuse - whether deliberate or not - of other features on StackExchange is met with negative feedback and punishment. Bad questions and answers get downvoted, deleted or closed, usually with an accompanying comment explaining why. This costs users rep and tells them what they did wrong. It is an inconsistency with the way that incentives work on the rest of the site that there is currently no punishment for misuse of the review tools, nor any feedback on whether your reviewing was useful.

When a post gets closed or deleted or ends up on negative score (without an intermediate edit), any reviewer who upvoted it should lose the review point they earned for doing so and suffer a 10 rep penalty in the bargain, with an explanation of the cause. That won't harm any diligent and honest reviewers - or will at worst do so only rarely when they make honest mistakes - but will quickly put an end to fake reviews of the kind described at length here:

The current review system encourages fake reviews; some people upvote everything rather than actually fixing problems

Answer 4

As far as I can tell, Late Answers queue makes a convenient tool to detect fake reviewers, by giving a sensible way to catch a verifiable abuse that is also free of undesired side effects.

• "verifiable abuse" is an upvote given to the post that was soon after that (eg in 24 hours) deleted as not an answer or as spam.
  For an example of abuse caught, refer here: three upvotes cast on blatant not-an-answer.

  

• "free of undesired side effects" means that downvoting answers costs a rep which would serve as a protection against fake reviewers that would want to thoughtlessly vote down instead of up to avoid being caught by suggested check.

handful reviews proven fake => two weeks review suspension

---

As someone who once has been a subject of similar measure for giving bad suggested edits, I can assure that week or two suspension is a very VERY educative measure.

---

Script similar to one that currently detects and reverts serial-voting can be used to catch those who abuse review. After all, brainless upvotes are just another kind of voting fraud aren't they?

Answer 5

The system is working as implemented.

SO lets people with enough rep review posts, but there is no training of any kind as to what is a good post and what is not. As a result you're getting who knows how many users going off and reviewing posts using their own unique rules and experiences.

Earlier I had asked here about how the decisions made on MSO were communicated to the broader community and the answer was basically "they're not". You want a self-moderating site but are unwilling to modify the site to enforce practices. As a result the only thing "the masses" see is a site where you can get points by doing certain things and the more points you have the more prestigious the profile.

These reviewers are doing exactly what you'd expect them to do.

Answer 6

It has been a long time since I was a regular on Slashdot, but I always found their metareviewing system very sensible, and maybe something similar would be a reasonable solution here.

At the moment, as far as I know, the only time a user's reviews are subject to meta-review by another editor is if they submit an edit that has to be reviewed and accepted. There are a few ways that you could construct a system such that good reviewers could receive reputation bonuses as an incentive to give their review actions a lot of thought, and that bad reviewers lose reputation (and eventually the right to review at all).

One possibility is a literal metareview: Let editors with a very high reputation score have access to a review of recent reviews, in which they simply agree or disagree with the reviewer's conclusions. The reviewer gets or loses 10 reputation based on the score, and if the reviewer has more than X reviews and more than N percent are voted down, they lose their ability to review.

Another possibility is to simply submit each review item to, say, 5 editors before removing it from the review queue. If 4 or 5 editors agree (maybe we categorize some actions as "good" and some as "bad" and consider two editors in agreement if their actions are in the same category) then they all get a bonus, and the odd man out gets the penalty. The disadvantage here would be that you could game the system by deducing that you're not the first editor to see an item and agreeing with whatever actions already took place, like by downvoting an item that's been downvoted twice already. But the current system can be gamed in the same way (especially the "test" items where you can look at the page, see that the item's been deleted, then downvote it and be congratulated for your cleverness).

Answer 7

The main reason for up voting is that some users need badges and reputaion.

That will happen when a new user got privilege for review

So the new user who got 125+ reputation has doing only 2 things

1)Up votes on answers

If he/she down voted loose reputation

2) up /down votes on questions

In my point of view there need a proper checking so that a particular review should be reviewed by atleast 5 users