Careless review upvotes are actively contributing to the quality problem, not correcting it

Question

Look at this reply. The question is from 2008, it has an accepted answer, and almost every other answer generally has the same theme and references the same function.

That reply, at the time of writing, has two upvotes. It is not an answer. It is blogspam, and it basically duplicates every other answer present while simultaneously proving no actual content. There is no way that this reply deserves two upvotes. It deserves to be nuked from orbit.

Now, look at this reply. Hit the revisions link, and look at the original. That unintelligible txtspeak crap received two upvotes at time of editing. I think I cleaned it up enough that it can stand on its own, but seriously, come on.

Now look at this reply. Again, it received two upvotes within minutes. It's not just crap, it's copied and pasted crap that included a typo from the question. There is no explanation of what it is, how it changed, how it's going to help, etc. It's a horrible answer and should not exist, and nobody in their right mind should ever have upvoted it.

How about this question. It isn't even on-topic, yet it has another two predictable upvotes.

I could go on and on, but I'm sure you're getting the point by now. That and many of my other examples are questions, and question downvotes aren't easily obtained.

Here's the bottom line. Users that are subject to review (via First Posts & Late Replies) are getting upvotes when they should not. Users that aren't subject to review are getting nothing. The default answer sort is by score. The crap is rising to the top. This sends bad messages on so many levels that it isn't even funny.

We have huge quality issues over in php, and the constant upvoting of crap puts the voting system in jeopardy, making it increasingly difficult to promote good content.

Answer 1

An optimization to the Late Answers, First Posts, and Low Quality review queues has been deployed that will keep a single review from being shown to multiple people at the same time. This should help the situation where multiple people would perform reviews which resulted in multiple up-votes. I will continue to monitor these queues closely.

Answer 2

This isn't really a permanent solution, but until we can work out the extent of the problem here - and a more automated solution - I'm manually suspending reviewing privileges for folks who fail multiple review audits in a short time, where exact definitions for "multiple" and "short" depend largely on overall activity.

Also, folks who are obviously cheating the audits will be blocked from /review for much longer periods of time.

Trivia: some of the folks involved are doing other dodgy things with voting as well.

Answer 3

Quick (and maybe drastic) fix: It should be not possible to cast a vote via /review. It should only be possible to do that when clicking off the review into the concrete post which is viewed "the usual way".

Reviewing is fixing/flagging posts, not voting posts you (dis)agree upon and/or found (un)useful. For that you'd have to browse the questions "the usual way".

Answer 4

The problem with careless upvotes (or careless downvotes/lack of feedback for that matter) is that it leads to a form of adverse selection (see for instance Akerloff (1970), "The market for Lemons").

The potential consequences are two-fold:

1. Distinguishing between low- and high-quality information becomes more difficult.
2. The market becomes inefficient and - in dramatic cases - may even fail (and SO is arguably a market, with knowledge being traded for reputation).

Rewarding low-quality (or not rewarding high-quality) is detrimental to SO because it devalues the platform itself. The incentive for providing high-quality information is reduced when the cost/effort-to-reward ratio is skewed towards low-quality.

The review queues exacerbate the problem because they provide an opportunity for questions and answers to receive additional attention - I'd argue that this further skews the market. The competitive element in the review process (where a posting can be simultaneously reviewed by different users) makes the matter worse because it favors the faster as opposed to the careful reviewer. Moreover, having different queues (first posting, late answers, etc.) introduces a selection bias.

That said, I agree in large part with Rober Harvey's analysis that "The problem as I see it has more to do with weaknesses in the voting system than it does with the review system's design". In an information market, askers create the opportunity for a trade - which places them in the role of a seller. As soon as a question is answered, the tables are turned: the answerer becomes the seller and competes against other answerers for the opportunity of an eventual reward in the form of a reputation gain at the risk of a potential reputation loss (downvoting).

One potential way to address the problem would be alter the review process by:

1. first of all merging the queues, providing a mix of old, current, first postings, late answers, etc. at random from the pool.
2. having upvotes and downvotes not affect the reputation score of a posting until a threshold x is reached. The threshold x would be dynamically calculated in function of the number of upvotes versus downvotes. Where the difference is not statistically different in one direction or another no score would be computed and the posting would be added to a queue for further review at a later time.

This would have the advantage of spreading the attention accorded to existing and new postings in a more uniform manner - leading to a hopefully better valuation of the existing information. Any takers?

Answer 5

It is important to shake things from time to time. But would be difficult to do it forever (considering growing user demographics). If a "shake up" results in any positive outcome one can still end up having only a temporary effect. In order to prolong this effect, we probably want to get to the situation where the following is sufficient:

Best form of moderation is self-moderation

I am a newbie here and I apologize for somewhat rhetorical tone. Nevertheless, I hope I am making a useful point here.

What I would discuss further probably can be described in a single sentence as user self-moderation and self-education by reading more meta Stack Overflow:

To answer the OP's question I started by sorting the PHP tag by votes and looked through the well-rated answers on well-rated questions, meaning that an analysis similar to OP's with an overall good Stack Overflow style and quality in mind can be applied there as well.

One still does find a lot of duplicates/reposts (e.g. this). Occasionally users give comments "Why this answer gets only X upvotes while almost a similar answer got Y (where Y >> X)?". It seems obvious that acting and thinking like this people do not prevent post duplication in a natural manner. Why? Is it because they don't care? Somehow I doubt that most of the users stay indifferent to Stack Overflow concepts while still actively making posts (at least not the entire Stack Overflow population). Next (arguable) guess would be that maybe users do not pay too much attention to the importance of following a good Q&A style because they have no idea about it :) ('said as a true captain')

Unfortunately, I don't think I am proposing an ultimate solution to the problem of a particular tag (php), but I would really like to share one simple (probably rather obvious) thought:

The more people read/participate in meta Stack Overflow, the more they think and learn about better practices of a good Q&A style. They become part of a community in yet another, additional way by facing problems from the conceptual or designing perspective.

Not sure how representative my case is, but I must confess that the more time I spend on reading meta Stack Overflow, the more I am getting into the problems discussed here - the more I care or get involved. In the past, I tended to play a role of just a "passive" guest, a visitor that comes by to get a desired piece of information. Not sure why, but it did take me a while to start paying greater attention to the meta part of SO.

I just want to say that even reading meta does makes a difference. And maybe one of the ways to improve overall writing quality is indeed to attract more people to meta in many possible ways.

By no means do I think that what I suggest here is something extraordinary or novel. I just merely confirm that it seems to be working, at least in my case.

From what I can tell, you guys already did a great job by making meta Stack Overflow more accessible to the visitor:

• There is a link to meta in the main menu (in the top navigation bar, in the bottom links, in the list of all of SE sites)
• Stack Overflow FAQ and community wiki pages are linked to meta
• Certain questions get migrated to meta (which redirect users).
• ...(something I don't know/can't think of right now)

The Stack Overflow site seems to do a lot to help to transit from a visitor to a registered user, and from occasional contributor to a standards-aware author. Doing so is an obvious motivation of improving and educating people. I guess a constantly pending question would be how to do it even better?

---

Finally, it could be not at all a bad idea to consider different types of writing Q&A standards to be tag-specific, custom enough (maybe including a possibility of extra obligatory input fields) to address the writing quality of a particular tag subgroup.

The best analogy I can provide is a bug reporting: better structured reporting form optimizes the testing cycle (requiring a screenshot, reproducibility instructions, software, and environment versions, etc.).