123

Look at this reply. The question is from 2008, it has an accepted answer, and almost every other answer generally has the same theme and references the same function.

That reply, at the time of writing, has two upvotes. It is not an answer. It is blogspam, and it basically duplicates every other answer present while simultaneously proving no actual content. There is no way that this reply deserves two upvotes. It deserves to be nuked from orbit.

Now, look at this reply. Hit the revisions link, and look at the original. That unintelligible txtspeak crap received two upvotes at time of editing. I think I cleaned it up enough that it can stand on its own, but seriously, come on.

Now look at this reply. Again, it received two upvotes within minutes. It's not just crap, it's copied and pasted crap that included a typo from the question. There is no explanation of what it is, how it changed, how it's going to help, etc. It's a horrible answer and should not exist, and nobody in their right mind should ever have upvoted it.

How about this question. It isn't even on-topic, yet it has another two predictable upvotes.

I could go on and on, but I'm sure you're getting the point by now. That and many of my other examples are questions, and question downvotes aren't easily obtained.

Here's the bottom line. Users that are subject to review (via First Posts & Late Replies) are getting upvotes when they should not. Users that aren't subject to review are getting nothing. The default answer sort is by score. The crap is rising to the top. This sends bad messages on so many levels that it isn't even funny.

We have huge quality issues over in , and the constant upvoting of crap puts the system in jeopardy, making it increasingly difficult to promote good content.

|improve this question
  • 20
    Welcome to the party. – Mysticial Dec 12 '12 at 9:03
  • 4
    Different angle. The other question is complaining about the way the reviewing system is set up. I'm complaining about the side-effects it's having on real content. – Charles Dec 12 '12 at 9:06
  • 3
    That too has also been somewhat beaten to death lately. It's all in the same pool of problems. – Mysticial Dec 12 '12 at 9:06
  • 41
    I agree with Mysticial, this doesn't bring anything new to the table. Still, any post to this effect gets a +1 from me. I do not understand why the review system is not turned off until the problem is sorted out. It is doing active, irreversible damage to the question pool every day. – Pekka supports GoFundMonica Dec 12 '12 at 9:07
  • 2
    Unfortunately I couldn't find anything substantial about this problem in the other question, only bits here and pieces there. I'm not going to be offended by dupe votes. But still. C'mon, this is ridiculous. – Charles Dec 12 '12 at 9:08
  • 9
    Please check the following link http://blah-blah It really helped me. Blatant case of link-only-answers gets two upvotes in two hours. O tempora o mores! This issue indeed deserves a dedicated discussion: "The default answer sort is by score. The crap is rising to the top. This sends bad messages on so many levels that it isn't even funny." – gnat Dec 12 '12 at 9:25
  • 4
    related (not a duplicate): This post was of very poor quality? Reviewer openly admits, "I have no technical knowledge of the topics... I tried to upvote." – gnat Dec 12 '12 at 10:02
  • 3
    I just found this one that got 4 upvotes in a matter of minutes in the First Post queue. It doesn't add anything to question. – Taryn♦ Dec 12 '12 at 12:02
  • 5
    It's pretty clear that the issues with upvoting are localized to two review queues: Late Answers and First Posts. That's why you see 2-3 upvotes for spam (10k links): stackoverflow.com/a/13567798/19679, stackoverflow.com/a/13450046/19679 and non-answers: stackoverflow.com/review/late-answers/1052069, stackoverflow.com/questions/6151164/user-input-in-a-matlab-gui/… . First posts by new users on old questions hit both of these review queues and get upvoted multiple times. I don't believe stray upvotes are a problem with the other queues. – Brad Larson Dec 12 '12 at 16:34
  • 2
    @BradLarson, that third link fills me with apoplectic rage. I've integrated your suspicions, as they seem spot on. Thank you. – Charles Dec 12 '12 at 17:42
  • 16
    how to get massive upvotes on meta: Step 1: complain about reviewers. – Sam I am says Reinstate Monica Dec 12 '12 at 17:49
  • 5
    @sam how does that explain 3 upvotes on an answer that is completely nonsensical (created by a bot), or spam? – Pekka supports GoFundMonica Dec 12 '12 at 18:58
  • 7
    @Sam there are plenty of spam or nonsense answers on SO getting upvoted since the review tool is in place. You are insinuating people are upvoting stuff innocently, because "they don't have the expertise to edit". How does that explain that complete, undisputable crap gets upvoted? – Pekka supports GoFundMonica Dec 12 '12 at 19:47
  • 15
    @Sam: "Upvoting a neutral question" is bad too (as is upvoting where you don't have expertise to tell the difference). We're not handing out gold stars for effort here. Only good things that contribute to SO's knowledge base should be upvoted. Neutral things that you can't improve yourself should be left for someone else. – jscs Dec 12 '12 at 20:00
  • 3
    @SamIam I agree with your general point about this having been done to death. We have a bunch of questions here that do nothing but point at instances of bad reviews, which is not something that is news to anyone on Meta at this point. That said, innocent upvotes for a crap post are just as destructive as "malicious" upvotes. The intention simply doesn't matter. – Asad Saeeduddin Dec 13 '12 at 8:02
26
+100

An optimization to the Late Answers, First Posts, and Low Quality review queues has been deployed that will keep a single review from being shown to multiple people at the same time. This should help the situation where multiple people would perform reviews which resulted in multiple up-votes. I will continue to monitor these queues closely.

|improve this answer
  • 2
    That sounds really good, thanks. In new deployment, did the delay for non-skip buttons remain as before, about 2 seconds? I ask because last time I tested "No Action Needed" button in First Posts queue, I measured that it took me less than half minute to click through about 10 posts. Realizing that this was about 5-10 times faster than I normally do when paying attention to post contents I thought that maybe current action delays give too much leverage to robo-clickers? – gnat Dec 20 '12 at 7:18
  • @GeoffDalgas - Over the last few weeks, I have noticed more and more First Posts, Late Answers and LQPs available for review on SO prime. Is this due to the optimization mentioned here? If so, it seems to have worked. Otherwise, some other magic is at work, and hooray for magic. – JoshDM Aug 12 '13 at 15:03
  • There are still complaints about it Ref. Is the feature working as expected or can it maybe get improved? – Trilarion May 14 '14 at 14:07
  • Can we add it to the Edit review cue as well, please. @Trilarion - your Ref is about the Edit Cue - I agree entirely. – simo.37920 Aug 15 '14 at 2:27
72
+50

This isn't really a permanent solution, but until we can work out the extent of the problem here - and a more automated solution - I'm manually suspending reviewing privileges for folks who fail multiple review audits in a short time, where exact definitions for "multiple" and "short" depend largely on overall activity.

Also, folks who are obviously cheating the audits will be blocked from /review for much longer periods of time.

Trivia: some of the folks involved are doing other dodgy things with voting as well.

|improve this answer
  • 3
    Would it be possible to get some aggregate data as to number of people reaching these thresholds, or perhaps the percentage of reviewers (reviewing more than X items) reaching these thresholds, to get some slightly more concrete info on the percentage of "bad" reviewers. – Servy Dec 12 '12 at 20:47
  • 26
    Quick stats: 348 users have failed at least one audit, 117 two, 36 four. That's out of 1471 reviewers who have been audited at least once. At four failures, I start looking closely - four in a week, very closely. – Shog9♦ Dec 12 '12 at 20:57
  • Thank you for the stats. – Servy Dec 12 '12 at 20:58
  • What sort of %age are we talking here? So are you saying that those 36 people have been suspended from the review queues? – Matt Dec 12 '12 at 20:58
  • 2
    @Matt: not all of them. Some of those failures go back a bit (into the period where we were still kinda hashing out what went into audits). So to avoid mistakes, I'm checking everything - it's a bit tedious. – Shog9♦ Dec 12 '12 at 21:04
  • (I'll post some better info in a bit; few too many irons in the fire) – Shog9♦ Dec 12 '12 at 21:05
  • 7
    @Shog9, the tedium is appreciated. Lemme know when you'll next be in Seattle, I'll buy you a beer or something. – Charles Dec 12 '12 at 21:05
  • 10
    @Shog9 - Thank you! It's finally happening! – hims056 Dec 14 '12 at 6:56
  • Hmm - I was NOT not paying attention so I was a little shocked – mplungjan Dec 18 '12 at 13:46
  • 1
    @Shog9 How can one 'obviously' cheat the audit? I do fairly few reviews, because the queue is almost always empty - but when I do review and I am in doubt ie. about the validity of an answer, I usually open the full question to see the context, see whether others have answered exactly the same, etc. When that happens it usually becomes clear if I'm in the middle of being audited. Would that be counted as a cheat? (If possible, I would think it should count as being thorough.) :-P – user213634 Jul 2 '13 at 14:58
  • 1
    I'd only suspect you were cheating if you were only that thorough for audits, @Anders. – Shog9♦ Jul 2 '13 at 16:04
  • Assuming the reviewer is actually reading the post they are reviewing... If suggested edit audits contain edits that are so obvious as these 2: stackoverflow.com/review/suggested-edits/3867154, stackoverflow.com/review/suggested-edits/3810598, how is it possible to ever fail an audit. Do audit posts typically contain edits which are so obviously "wrong"? – Kevin Fegan Jun 22 '15 at 22:39
16

Quick (and maybe drastic) fix: It should be not possible to cast a vote via /review. It should only be possible to do that when clicking off the review into the concrete post which is viewed "the usual way".

Reviewing is fixing/flagging posts, not voting posts you (dis)agree upon and/or found (un)useful. For that you'd have to browse the questions "the usual way".

|improve this answer
  • 4
    Totally agree. If you find a question interesting enough to want to upvote it, you won't mind opening a new tab. – Asad Saeeduddin Dec 16 '12 at 11:20
  • 1
    No, I want to be able to do my voting without necessarily having to click into the post. We need a different solution than crippling the interface. – Lance Roberts Dec 19 '12 at 18:34
  • 1
    I agree, you shouldn't be voting without seeing the post in context. – Ben Voigt Feb 7 '13 at 22:58
5

The problem with careless upvotes (or careless downvotes/lack of feedback for that matter) is that it leads to a form of adverse selection (see for instance Akerloff (1970), "The market for Lemons").

The potential consequences are two-fold:

  1. Distinguishing between low- and high-quality information becomes more difficult.
  2. The market becomes inefficient and - in dramatic cases - may even fail (and SO is arguably a market, with knowledge being traded for reputation).

Rewarding low-quality (or not rewarding high-quality) is detrimental to SO because it devalues the platform itself. The incentive for providing high-quality information is reduced when the cost/effort-to-reward ratio is skewed towards low-quality.

The review queues exacerbate the problem because they provide an opportunity for questions and answers to receive additional attention - I'd argue that this further skews the market. The competitive element in the review process (where a posting can be simultaneously reviewed by different users) makes the matter worse because it favors the faster as opposed to the careful reviewer. Moreover, having different queues (first posting, late answers, etc.) introduces a selection bias.

That said, I agree in large part with Rober Harvey's analysis that "The problem as I see it has more to do with weaknesses in the voting system than it does with the review system's design". In an information market, askers create the opportunity for a trade - which places them in the role of a seller. As soon as a question is answered, the tables are turned: the answerer becomes the seller and competes against other answerers for the opportunity of an eventual reward in the form of a reputation gain at the risk of a potential reputation loss (downvoting).

One potential way to address the problem would be alter the review process by:

  1. first of all merging the queues, providing a mix of old, current, first postings, late answers, etc. at random from the pool.
  2. having upvotes and downvotes not affect the reputation score of a posting until a threshold x is reached. The threshold x would be dynamically calculated in function of the number of upvotes versus downvotes. Where the difference is not statistically different in one direction or another no score would be computed and the posting would be added to a queue for further review at a later time.

This would have the advantage of spreading the attention accorded to existing and new postings in a more uniform manner - leading to a hopefully better valuation of the existing information. Any takers?

|improve this answer
  • 2
    The ideas seem interesting but the two ways you state I think are wrong. 1) The Close and reopen vote and edit queues are different to the rest as these questions can be looked ato their own, the others just do not have enough info on the review page and so take a lot longer to do properly and you should skip more of these if you do not know the subject or many just are OK and you can't change. (Disclosure I have Steward flags for close and edits but not the others) and 2) the reputation of the posting is not the issue but the rewards for reviewing. – user151019 Dec 16 '12 at 11:35
  • @Mark thanks for the feedback but 1) my post did not address close/reopen vote and edit queues - rather I focused on the queues I have access to where I have noticed what looks like bling upvoting patterns (one of the reason why I seldom review). 2) to my way of thinking the upvotes/downvotes cast during reviews are an essential part of the problem with the process because they affect the signal to noise ratio - in other words they directly influence the perceived worth of the reviewed posts when accessed by users later. The rewards for reviewing represent a different aspect of the problem. – Anthill Dec 16 '12 at 11:48
4

It is important to shake things from time to time. But would be difficult to do it forever (considering growing user demographics). If a "shake up" results in any positive outcome one can still end up having only a temporary effect. In order to prolong this effect, we probably want to get to the situation where the following is sufficient:

Best form of moderation is self-moderation

I am a newbie here and I apologize for somewhat rhetorical tone. Nevertheless I hope I am making a useful point here.

What I would discuss further probably can be described in single sentence as user self-moderation and self-education by reading more meta SO:

To answer the OP's question I started by sorting php tag by votes and looked through the well rated answers on well rated questions, meaning that an analysis similar to OP's with an overall good SO style and quality in mind can be applied there as well.

One still does find a lot of duplicates/reposts (e.g. this). Occasionally users give comments "Why this answer gets only X upvotes while almost a similar answer got Y (where Y >> X)?". It seems obvious that acting and thinking like this people do not prevent post duplication in a natural manner. Why? Is it because they don't care? Somehow I doubt that most of the users stay indifferent to SO concepts while still actively making posts (at least not entire SO population). Next (arguable) guess would be that maybe users do not pay too much attention to the importance of following a good Q/A style because they have no idea about it :) ('said as a true captain')

Unfortunately, I don't think I am proposing an ultimate solution to the problem of a particular tag (), but I would really like to share one simple (probably rather obvious) thought:

The more people read/participate in meta stackoverflow, the more they think and learn about better practises of a good Q/A style. They become part of a community in yet another, additional way by facing problems from the conceptual or designing perspective.

Not sure how representative my case is, but I must confess that the more time I spend on reading meta SO, the more I am getting into the problems discussed here - the more I care or get involved. In the past I tended to play a role of just a "passive" guest, a visitor that comes by to get a desired piece of information. Not sure why, but it did take me a while to start paying greater attention to the meta part of SO.

I just want to say that even reading meta does makes a difference. And maybe one of the ways to improve overall writing quality is indeed to attract more people to meta in many possible ways.

By no means do I think that what I suggest here is something extraordinary or novel. I just merely confirm that it seems to be working, at least in my case.

From what I can tell, you guys already did a great job by making meta SO more accessible to the visitor:

  • There is a link to meta in the main menu (in the top navigation bar, in the bottom links, in the list of all of SE sites)
  • SO FAQ and community wiki pages are linked to meta
  • Certain questions get migrated to meta (which redirect users).
  • ...(something I don't know/can't think of right now)

SO site seems to do a lot to help to transit from a visitor to a registered user, and from occasional contributor to a standards-aware author. Doing so is an obvious motivation of improving and educating people. I guess a constantly pending question would be how to do it even better?

Finally, It could be not at all a bad idea to consider different types of writing Q/A standards to be tag specific, custom enough (maybe including a possibility of extra obligatory input fields) to address writing quality of a particular tag subgroup.

Best analogy I can provide is a bug reporting: better structured reporting form optimizes testing cycle (requiring a screenshot, reproducibility instructions, software and environment versions, etc.).

|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .