What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

I found good questions about this subject (like that one) but they weren't quite narrowing to what I wanted to ask. So I wrote my own.

The problem

I have been reviewing for a few months now (~725 reviews) in those two sections (Late Answers and First Posts) and I have seen so many upvote on question that doesn't really deserved it. (e.g. Here and here)

In those case, I was almost angry about the reviewers that upvote them, feeling that they didn't took anytime to understand what's the question, which answers are already there, is the question already solved, how much the post they are reviewing is adding value to the thread, etc.

It happens often and during I'm reviewing posts. I'm still reading through and suddenly 1 then 2 and even 3 upvotes in a row.

I think some user don't put enough time in a review, maybe the Upvote option is so easy or at some point they don't really bother to read it entirely and they just want to get their daily cap.

In depth analysis of the situation

Maybe they don't understand the impact of a bad review on the community on the long term. The thing is those First Posts and Late Answers are the best way to educate new users to SO (or any other) and explain them how it works.

If people just up-vote them quickly without putting enough time evaluating which action would be best, we will lose the opportunity to show to those new users how it works.

When they got upvote 2-3 times for a bad answer, for instance, they will think it's the way to write them. They will come back later posting similar bad answer elsewhere. However, that time the reviewing system won't catch it!

I know there is a limit on the number of posts you can review per day. There is also if you reached your daily vote limit. Both helps, but I don't find it enough to prevent those kind of bad reviews.

Some suggestions

I thought about few things, but I'd like to know what other people think about this.

Limit the number of Upvote in the context of a review

For example, if you are allowed to review 20 Late Answer and 20 First Post a day, you have a limit of 10 Upvotes. Think about it, how many times half the reviewing post will really deserve an upvote? You're are lucky if you got 3 in a row IMO.

Find a way informs users how to perform a better review

Maybe there is not enough emphasis about how to make a good review? The first time they access the Review system, we could show them a page to teach them what is a good and a bad review.

Increase the reputation needed to make those review to 500

It honestly took me a little while to understand a lot of aspects of the system : what is the mentality when reviewing, what should be done. I often had questions myself and I went on Meta reading threads to know what should I do in some cases. For instance, when the answer is very similar to another one, which flag I should use, etc.

I don't think a lot of people actually do that. That's why I think there should be a lower limit of 500 rep, so users have enough experience to make proper review. Because even though they are the first type of review users can do, they are important IMO.

EDIT :

I don't want to target anyone, but this enforce my point. I have seen users (member since 2-3 months) with more than 2000 reviews done in total and they have literately ~1 500 vote cast where only ~10 are voted down. I highly doubt that so many reviewed posts were good enough to be up-voted. The saddest part of this, is that even with a limit of 1k rep (as suggested by Kevin), some of them would not be affected by that limit.

share|improve this question
3  
I'd be inclined to say 1k or a S&W badge. –  Kevin Nov 10 '12 at 22:08
    
@Kevin I thought about that too, recently I noticed that they increased the limit from 1k to 2k and 2k to 3k for the other type of review tasks. It might be a good idea to do the same in that case. It's just important to keep a certain progression in my opinion. I thought 500 was a good start, because some people could be good reviewer at 200 some other at 800. –  ForceMagic Nov 10 '12 at 22:10
1  
Yes, and that's why I said S&W. Now, the way people approve edit suggestions on SO it won't guarantee good reviewers, but it at least strongly indicates a desire to improve the community. Recall that under the old system, S&W was required to review at all. –  Kevin Nov 10 '12 at 22:17
    
Yep, I've seen that before and I thought it make sense. But I guess they might have wanted to delegate some of the review types to newer people to unload high rep users. –  ForceMagic Nov 10 '12 at 22:20
1  
It's often when people are below 2K, since when you reach 2k you can edit a post directly. I hope when they will reach 2k, they will start to review and not upvote. I'm 100% with you, I hate this kind of bad review. –  j0k Nov 15 '12 at 18:27

7 Answers 7

up vote 5 down vote accepted

update

Equally easy but hopefully less damaging review option has been introduced.

"No Action Needed" button has been added in First Posts and Late Answers review queues. It increases review count but does not require reviewer to vote.

http://i.stack.imgur.com/Ab5MC.png

share|improve this answer
1  
Awesome :) Another thing that could be added is the exclusive review. I don't know if it's in the plan as well, maybe that should be discussed in another thread, but it is also annoying to start a long but good edit and the realise that someone else posted a smaller one. Any plans on this? Thanks anyways :) –  ForceMagic Dec 15 '12 at 9:26
    
@ForceMagic well fully exclusive review would be potentially dangerous because of unlimited blocking: one very slow reviewer could completely block others acting on that item. But some balanced sort of it has been proposed here: "This would guarantee that my decisions (at least quick ones) could only clash with those of the user(s) who held the suggested edit longer than mentioned timeout - which would weed out mindless click-through robo approvers...." –  gnat Dec 15 '12 at 9:44

I quietly suspect that the root of the problem is that a large and well-understood part of the purpose of Review is to check for a few fairly objective Bad Things - including, but not limited to, spam, non-questions, and non-answers - but if none of those issues are present, there's no way of simply saying "Nothing is horribly wrong here" except an upvote.

I am a quite inexperienced programmer, and very frequently in review I'm faced with a question or answer that I can see is written in clear and correct English, is not spam, and is not patently idiotic... but it is in a field that I know nothing whatsoever about, so I can't reliably discern much more than that. I'm often tempted to upvote these just as a way of signalling "Everything looks okay here", but instead usually click 'Skip' instead - often after having invested minutes of my time researching around the subject matter, but without learning enough to be confident that an upvote is appropriate. This is really off-putting. It leaves me knowing that the minutes of time I've spent reviewing that question will serve no purpose for anyone.

But then a doubt comes into my mind: maybe I should just upvote? I vaguely understand the question; it appears reasonable; from his writing style, the OP seems to be sensible and know what they're talking about; I'd be willing to bet that an upvote is appropriate here. Why let my careful reading and thought go to waste, especially when I notice that three other people have smacked the upvote button while I've been pondering the question? Basically, the whole system at present encourages me to click the upvote button when I'm in doubt.

This experience makes me suspect that many of the upvotes on first posts that people on meta tend to attribute to bad people 'gaming' the review system for badges are really just down to people honestly upvoting questions they don't understand, but which appear, superficially, to be reasonable questions.

I wonder if the best mitigation to the problem is not to make upvoting more difficult, but to provide an equally easy alternative that doesn't send out the potentially misleading signals that an upvote does. Why not simply have an Everything looks fine here checkbox that you could check to let you click the I'm Done button without having to upvote the post you were reviewing? Honest reviewers would be able to indicate that a post appears harmless without having to (perhaps inappropriately) upvote it, the lazy and unscrupulous badge farmers could still lazily and unscrupulously badge farm but in a quieter way that would cause less collateral damage for the site, and everyone would be mildly happier than they are now.

share|improve this answer
1  
Indeed, that's also why I linked this thread : meta.stackexchange.com/questions/147867/… in my question. Maybe a that solution of those "Looks fine" button would help a bit. But I still some people just UpVote without really reading the post like you do. It kinda frustrates me when it happens while I'm writing a comment to explain to the user what could make his post more valuable. That happens when I find that his post is not bad, but could be improved a lot. –  ForceMagic Nov 13 '12 at 0:56
    
This is how it should be. Take a look at my argument on another question. –  animuson Nov 13 '12 at 16:12
    
@ForceMagic - you're quite right. I haven't really addressed the point you were raising here and I would downvote my own post now if I could. The lack of a "Looks fine" button is one issue that I think ought to be fixed, but it's distinct from the problem of obviously fake reviews - which I've realised after reading around some more is a real and widespread one. I'm about to post another answer that I think responds to that problem more directly. –  Mark Amery Nov 13 '12 at 19:12
    
Awesome! Thanks for your concern and professionalism. (I did upvote that answer thought, because the "Looks fine" button may help in that situation as well). –  ForceMagic Nov 13 '12 at 19:16

If reviewers are upvoting questions or answers that have serious, visible problems - ones that get closed or deleted (for any reason other than being duplicates or deletion by the original poster) - then they're not doing their job and we should punish them for it.

Misuse - whether deliberate or not - of other features on StackExchange is met with negative feedback and punishment. Bad questions and answers get downvoted, deleted or closed, usually with an accompanying comment explaining why. This costs users rep and tells them what they did wrong. It is an inconsistency with the way that incentives work on the rest of the site that there is currently no punishment for misuse of the review tools, nor any feedback on whether your reviewing was useful.

When a post gets closed or deleted or ends up on negative score (without an intermediate edit), any reviewer who upvoted it should lose the review point they earned for doing so and suffer a 10 rep penalty in the bargain, with an explanation of the cause. That won't harm any diligent and honest reviewers - or will at worst do so only rarely when they make honest mistakes - but will quickly put an end to fake reviews of the kind described at length here:

The current review system encourages fake reviews; some people upvote everything rather than actually fixing problems

share|improve this answer
3  
I think that's a very valuable answer. I can remember easily many times where I saw people Upvote an answer while I was flagging it at the same time. And they were finally deleted. It must have some sort of check to ensure they loose Rep and points only if it was deleted by a flag though. Some user delete their own post, which is different. –  ForceMagic Nov 13 '12 at 19:20
    
@ForceMagic Good point. Edited my answer to include it. –  Mark Amery Nov 13 '12 at 19:22
    
Well, for starters, this system is easily gamed. If someone is potentially punished for upvoting everything they'll just start downvoting everything instead, as an example, upvoting a comment, or making an edit that doesn't change anything (meaningful), posting a spam comment, etc. –  Servy Nov 13 '12 at 19:22
    
@Servy Downvoting already costs rep whether it's justified or not. –  Mark Amery Nov 13 '12 at 19:23
    
@Servy Downvoting will cost them Reputation though (except for question). But maybe a combinaison of this and a "Looks Fine" button could solve the problem. –  ForceMagic Nov 13 '12 at 19:23
    
@ForceMagic If you add a "looks good" then they'll just "looks good" everything, as happens in the low quality posts queue. –  Servy Nov 13 '12 at 19:24
1  
@MarkAmery not for questions. –  Servy Nov 13 '12 at 19:25
    
@Servy Could the same process could be applied there then? If they click "Looks Good" but the post get deleted, they loose Rep. Because they should have clicked on "Skip" instead. –  ForceMagic Nov 13 '12 at 19:26
    
@ForceMagic Well, most posts in those queues (other than low quality posts) shouldn't be deleted; most of them just need to be edited into better shape. If you only do anything on deletion you are affecting too few posts for people to care when they get it wrong. And again, they can always downvote all questions, post a spammy comment, upvote any existing comment, make a meaningless edit, etc. –  Servy Nov 13 '12 at 19:27
    
@Servy What do you propose then? –  ForceMagic Nov 13 '12 at 19:31
2  
@Servy Even if they do click "Looks good" on everything, that's still less harmful than them upvoting everything. I agree it's not an entirely satisfactory solution, but do you not think that the suggestions I've made would at least be significant steps in the right direction? –  Mark Amery Nov 13 '12 at 19:31
    
@ForceMagic Well, a good place to start would be reading my answer on the question linked in this one; I've laid out several of my views there. –  Servy Nov 13 '12 at 19:34
    
@MarkAmery it would do slightly less harm, maybe, but I'd prefer fixing the underlying issues. If most everyone is just "looks good"-ing everything, we may as well just remove the review queues entirely as they'll have no effects at all. –  Servy Nov 13 '12 at 19:35
    
@Servy I understand your views and you state few good points on that post. However, I don't see any actionable suggestions concerning First Posts and Late Answers. –  ForceMagic Nov 13 '12 at 19:44
3  
@ForceMagic Remove the incentives to review. Take away the badges and the leaderboards. Don't keep track of the number of reviews, or at least don't make it easily publicly accessible information. I'd also like to see a good meta post with a "how to review" guide, but I don't discuss that in that answer much. –  Servy Nov 13 '12 at 19:48

As far as I can tell, Late Answers queue makes a convenient tool to detect fake reviewers, by giving a sensible way to catch a verifiable abuse that is also free of undesired side effects.

  • "verifiable abuse" is an upvote given to the post that was soon after that (eg in 24 hours) deleted as not an answer or as spam.
    For an example of abuse caught, refer here: three upvotes cast on blatant .

    http://i.stack.imgur.com/PMz1X.png

  • "free of undesired side effects" means that downvoting answers costs a rep which would serve as a protection against fake reviewers that would want to thoughtlessly vote down instead of up to avoid being caught by suggested check.

handful reviews proven fake => two weeks review suspension


As someone who once has been a subject of similar measure for giving bad suggested edits, I can assure that week or two suspension is a very VERY educative measure.

http://i.stack.imgur.com/XJWwR.png


Script similar to one that currently detects and reverts can be used to catch those who abuse review. After all, brainless upvotes are just another kind of voting fraud aren't they?

http://i.stack.imgur.com/tsE6O.png

share|improve this answer

The system is working as implemented.

SO lets people with enough rep review posts, but there is no training of any kind as to what is a good post and what is not. As a result you're getting who knows how many users going off and reviewing posts using their own unique rules and experiences.

Earlier I had asked here about how the decisions made on MSO were communicated to the broader community and the answer was basically "they're not". You want a self-moderating site but are unwilling to modify the site to enforce practices. As a result the only thing "the masses" see is a site where you can get points by doing certain things and the more points you have the more prestigious the profile.

These reviewers are doing exactly what you'd expect them to do.

share|improve this answer
2  
Minor nitpick... reviewing can increase your badge count, but not your reputation. –  Pops Nov 13 '12 at 19:26
1  
@PopularDemand You can consider reputation in the general sense, rather than reputation in the SE sense. For example, being on the leaderboard for a review queue can lead to people recognizing your name, thus increasing your reputation in the general sense. –  Servy Nov 13 '12 at 19:36

It has been a long time since I was a regular on Slashdot, but I always found their metareviewing system very sensible, and maybe something similar would be a reasonable solution here.

At the moment, as far as I know, the only time a user's reviews are subject to meta-review by another editor is if they submit an edit that has to be reviewed and accepted. There are a few ways that you could construct a system such that good reviewers could receive reputation bonuses as an incentive to give their review actions a lot of thought, and that bad reviewers lose reputation (and eventually the right to review at all).

One possibility is a literal metareview: Let editors with a very high reputation score have access to a review of recent reviews, in which they simply agree or disagree with the reviewer's conclusions. The reviewer gets or loses 10 reputation based on the score, and if the reviewer has more than X reviews and more than N percent are voted down, they lose their ability to review.

Another possibility is to simply submit each review item to, say, 5 editors before removing it from the review queue. If 4 or 5 editors agree (maybe we categorize some actions as "good" and some as "bad" and consider two editors in agreement if their actions are in the same category) then they all get a bonus, and the odd man out gets the penalty. The disadvantage here would be that you could game the system by deducing that you're not the first editor to see an item and agreeing with whatever actions already took place, like by downvoting an item that's been downvoted twice already. But the current system can be gamed in the same way (especially the "test" items where you can look at the page, see that the item's been deleted, then downvote it and be congratulated for your cleverness).

share|improve this answer

The main reason for up voting is that some users need badges and reputaion.

That will happen when a new user got privilege for review

So the new user who got 125+ reputation has doing only 2 things

1)Up votes on answers

If he/she down voted loose reputation

2) up /down votes on questions

In my point of view there need a proper checking so that a particular review should be reviewed by atleast 5 users

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .