-14

Introduction

As part-sociologist, I have been thinking about this for a while. It seems to me that, while voting is superficially a good way to measure community influence, its prominence on the questions and answers they prefix tends to lead to self-reinforcing, illogical voting patterns--which of course undermines the whole point.

The reason is something called pre-condition bias (a related term is anchoring). Essentially, the current score of a question or answer influences how people receive it in a positively correlated way.

The practical upshot is that downvoted posts automatically go further down and upvoted posts automatically go further up, neither with much bearing on posts' quality.

Let's look at some examples, and then I will present a proposed solution.


Motivation

Here's an answer of mine on english.stackexchange. Analysis (blockquoted for clarity):

My answer has 46 words, of which 21 come from a dictionary definition and another 7 are quoted from someone else. To write this answer, all I had to do was remember a term, look it up, and copy+paste. It took me all of five minutes. As of today, it has 61 upvotes.

After I posted it, I got 10 upvotes all pretty much at once, and from there, from its cushy location at the top of the answer pile, it just kept gathering more. There's no way I deserved that for 46 words.

The fact is, this is just an average post. It's a simple answer to a simple question, and a fantastic answer it ain't.

Contrast this with my recent answer on stackoverflow. Analysis:

As of today, it has 19 downvotes (and 10 up). How it got to be this way is fascinating. I posted the first answer and got an immediate upvote. Then, someone thought I ought to mention the effect on object-oriented code, downvoted, and left a comment. Though the answer fluctuated around 0 for a few hours, the moment it passed -2 on its Gaussian walk, everyone started downvoting. What started out as a fairly even balance of 8 up vs. 10 down suddenly became 10 up vs. 19 down!

What this says to me is that past -2 votes, the negativity of the average voter "just happened" to increase by 279%. Of course, then came the comments decrying the answer. And at least one of the comments I wrote in my defense "somehow" got deleted.

Meanwhile the question, inundated with attention, acquired 47 upvotes. But this question clearly had no research effort. If you Google its title, the first three hits are exit(int) (the call I suggested), a post from 2008 which advocates exit(int), and a duplicate on stackoverflow whose accepted answer suggests exit(int).

By the way, the accepted answer on that duplicate question is functionally identical to mine, and yet it has 43 upvotes to my -9.

The extreme examples are amazing:

Look at this one sentence question with more than two thousand upvotes. Is that really three orders-of-magnitude better than this nice question? The first user has posted just 33 times in total on the entire StackExchange, but because of that one sentence he now has moderation tools.

Look at this one which states a question on line 1, describes steps taken to solve, and provides a simple example, yet somehow has -44 votes (spoiler: it got some negative publicity on meta). Is that post, denigrated for grammar and sass, really forty times worse than this semi-incoherent drivel?


What's happening in all these examples is that votes have an inflationary effect. If once a post goes positive, it gets higher faster. If once a post goes negative, it gets lower faster. I know we're supposed to judge posts on their merits, but this sociological precondition bias effect is very, very powerful--and it is a sad reality that votes on StackExchange are almost never merit-based beyond the first couple of votes in one direction.

I believe this is a serious flaw in the way StackExchange operates. Posts on StackExchange are almost invariably direct functions of their initial votes multiplied by pageviews. I challenge the community to find a majority--or even a significant minority--of posts that defy this.


Towards a Solution

But this wouldn't be much more than a complaint unless I had an idea for how to fix it.

Reputation comes in several varieties:

  1. On users, it serves to delegate or restrict privileges. The higher the reputation, ostensibly the more trust the community places in them.
  2. On users, it is a rough measure of how good the quality of an answer or question is expected to be (and how much the user understands the StackExchange system). It's a point of respect. It helps you know how invested the user is in the community.
  3. On questions and answers (via votes), it (theoretically) serves to order posts by quality.

All of these are well-thought-out, important aspects, and any solution should preserve them.

But there's an important thing to note in the above points: reputation is nothing more than imaginary internet points--a proxy for worth to the community. When a question is posted or an answer is given, the differential value readers assign to it should not have any bearing on other readers, nor should it unduly and recursively influence the poster's privileges. What I mean is that reputation changes are supposed to affect standing in the community, not bias readers in reading that user's content.


So here's what I propose:

  1. At the very least look into this problem with an eye toward possible fixes. I believe this is a serious flaw that at minimum deserves attention from the community.
  2. Consider my pet idea: make votes on questions and answers on non-meta sites completely invisible to all but the poster and moderators.

    As a longtime StackExchange user myself, I understand that this would unquestionably be a radical change--probably one that would have to be tested in a limited rollout on one or a few networks. Actually, I think it would fundamentally alter the way the site works.

    But think about what we stand to gain. It solves the majority of the problem. Votes still are accrued by post, and affect the poster's value to the community. However, the reader would be unable to ascertain the degree to which the post is endorsed or reviled beyond the difference one vote makes to relative ordering of answers. This would leave the reader to judge the content primarily on its own merits.

    And isn't that what we want?
  3. Regardless of the solution ultimately chosen, the StackExchange voting system is utterly broken because of this biasing effect. This needs to be fixed yesterday.

Thank you for your consideration.

  • 5
    xkcd.com/552... just because one thing happens, and then another thing happens, it doesn't mean the former caused the latter. – hichris123 May 16 '15 at 17:01
  • 1
    The purpose of SO is to provide answers to questions and information to readers. Do you have actual evidence to suggest that otherwise valid information is becoming invalidated by the display of votes? Why do you believe that inflated vote counts affect quality of information and ability of readers to gain benefit from the site? Do you believe that a reader's ability to read an answer, try it, and say "this solved my problem" or "this did not solve my problem" is somehow diminished by visible vote counts? Because that's all that really matters. – Jason C May 16 '15 at 17:02
  • @JasonC as I wrote, reputation is a proxy for community standing, mapping directly to privileges. The point is that it fails at that aim. Instead, it is more like exp(randsgn()*pageviews). – imallett May 16 '15 at 17:05
  • 2
  • @imallett And as I wrote: So? Why is that a problem? What negative effect is it having on the purpose and goal of the sites? Your motivation section states that a thing happens but doesn't identify why it's a problem. We already know the reputation system isn't the most accurate way to map to privileges, if that's what you're aiming for, but it's decent enough. In any case, can you answer any of the questions in my previous comment? – Jason C May 16 '15 at 17:07
  • 1
    @JasonC the point is there's almost no relationship. Inflated votes aren't ipso facto a problem; the problem is that they're inflating something that's essentially random. For example, how many people are going to try a -8 voted answer? How many people going to scroll past a +100 answer to find a better one? And giving moderator privileges to lucky newbies who happen to ask basic questions at the right time, without actually being invested in their own community seems like a bad idea too. And again, I cannot stress this enough: it causes people to judge posts on votes, not merit. – imallett May 16 '15 at 17:15
  • Votes aren't essentially random, and you also haven't provided nearly enough evidence in your few cherry-picked examples to show that vote inflation happens (and is an issue). I don't think bandwagoning happens as much as you think on these sites. For example, "how many people are going to try a -8 voted answer" - you're forgetting a massive point here: You're forgetting that if that answer is actually correct, that at least 8 or more of the hundreds of thousands of involved and knowledgeable users on the site who may see that answer and review (but not need to try) it upvote it. Etc. – Jason C May 16 '15 at 17:18
  • That is to say, far more people vote on questions and answers than just people who happened to find the question in a google search because they were looking for an answer. The active members of this community review the existing questions and answers. That's the strength of these sites in general. (As for your scrolling past a +100 answer, yeah, they will scroll past it if it doesn't answer the question. They have no reason to scroll past it if it's the correct answer. No issue with information quality there.) – Jason C May 16 '15 at 17:19
  • 2
    Anchoring is a strongly established phenomenon, and I'm pretty damn certain that it had a significant effect on votes, but it would be really cool and insightful for stack exchange to perform some experiments to measure exactly how large the effect is. I don't think there's much that could be done to improve it without negative side effects, but if the effect turned out to be stronger than anybody would expect, maybe it would be worth considering. – Jeremy Banks May 16 '15 at 17:49
  • 2
    I am extremely confused. Did you get your references mixed up? You mention about your -9 SO answer that is functionally identical to the accepted answer with 43 upvotes. The post you linked does have a -9 score, but the accepted answer has a lot more than 43 upvotes and seems to say the exact opposite of yours. – psubsee2003 May 16 '15 at 17:51
  • @JeremyBanks Giving the benefit of the doubt and assuming anchoring / bandwagoning had a significant effect on votes: Why is that a problem? – Jason C May 16 '15 at 17:54
  • @psubsee2003 on the duplicate question. Edited for clarity. – imallett May 16 '15 at 18:26
  • 1
    I would say that anchoring is the least thing that happened with your examples (assuming that it has as large an effect as you presuppose). – Oded May 16 '15 at 18:53
  • 2
    Your answer is bad. The downvotes poured down when the question got bumped by being in the Hot Network Questions list, drawing thousands of views. No effect whatsoever - just some people who care doing the right thing. I find it amazing that instead of admitting the fact you gave a bad answer, you try to find excuses and other things to blame. – Shadow9 May 16 '15 at 19:10
  • 2
    Your biggest problem is you are trying to compare votes on your answer to votes on another answer in a completely different question (even one that is a duplicate). It is already know that voting between questions are skewed based on visibility (was it tweeted? did it end up on Reddit? was it on the hot question list?) and time asked (questions asked when Europe and America is sleeping get fewer views than during the work-days of both regions) and date asked (older questions get more votes for simply being old as well as being asked when question & voting standards were different. – psubsee2003 May 16 '15 at 20:32
7

I have to apologize in advance, this is very disorganized. Hopefully, a clearer answer will be posted.

First, commenting on your specific examples:

A word that means suffering great loss if failed but highly profitable if successful?

My answer has 46 words, of which 21 come from a dictionary definition and another 7 are quoted from someone else. To write this answer, all I had to do was remember a term, look it up, and copy+paste. It took me all of five minutes. As of today, it has 61 upvotes.

After I posted it, I got 10 upvotes all pretty much at once, and from there, from its cushy location at the top of the answer pile, it just kept gathering more. There's no way I deserved that for 46 words.

The fact is, this is just an average post. It's a simple answer to a simple question, and a fantastic answer it ain't.

That's how it works on that site. You do deserve the votes you got because it is a helpful answer to a common question -- it's more than just an average question. The number of votes on your answer is of the same order of magnitude as the number of votes on the other answers there. Many people apparently have that question, many people found your answer helpful, and so you provided a good answer to a common question (currently that question averages around 30-40 views per day). The votes reflect that, as they should.

If anything you should feel proud of providing a decent answer to a common question. Just because it's obvious doesn't mean it's bad. We all act as a Google Relay Server every once in a while, and that's OK. You gave a good answer. Thanks for that, and please use the privileges you gain from providing good answers for the betterment of the site.

How to end C++ code

As of today, it has 19 downvotes (and 10 up). How it got to be this way is fascinating. I posted the first answer and got an immediate upvote. Then, someone thought I ought to mention the effect on object-oriented code, downvoted, and left a comment. Though the answer fluctuated around 0 for a few hours, the moment it passed -2 on its Gaussian walk, everyone started downvoting. What started out as a fairly even balance of 8 up vs. 10 down suddenly became 10 up vs. 19 down!

What this says to me is that past -2 votes, the negativity of the average voter "just happened" to increase by 279%. Of course, then came the comments decrying the answer. And at least one of the comments I wrote in my defense "somehow" got deleted.

You haven't shown a link between vote count and vote behavior. If anything this suggests a link between a comment and vote activity, and a weak link at that. Even more, it shows a link between the addition of a much better answer that made casual readers realize yours wasn't that great and votes. More convincingly, this simply shows a link between time and stability of votes. It shows that it took a few minutes for enough people to view the answer to have the votes reflect that your answer wasn't that great.

Votes have a lot of initial noise. I posit that your answer deserves the votes it currently has, and that you were misled by the initial up/down noise. Over time it evens out. The duplicate question you mentioned gained 43 upvotes over five years and 130k views; this does not sufficiently demonstrate that it is a strong answer in comparison to yours, which is currently one day old. In fact, the old question you identified has now been marked as a duplicate of the new one, which provides a better answer, which has already far exceeded the number of upvotes on the old answer. The old answer simply wasn't that great, and this is, in a way, a counter example to your claim but across multiple questions instead of a single one: Having a one-day-old answer exceed a five-year-old answer's vote counts and having the old question marked as a duplicate of the new one is almost the polar opposite of vote inflation, and illustrates the point I mentioned above of a newer answer changing the landscape and affecting the voting on older answers.

In any case, continue answering questions, this one question is not statistically significant enough to determine your overall community standing; looking at your large collection of other decent answers on SO, I would say your reputation overall is a decent indicator of your general bigger picture standing, regardless of this specific answer.

Actually, a similar thing just happened to me, although I was on the opposite end. A question on Super User was asked and an incorrect answer based on common misconceptions was given then accepted. The answer garnered about 5-10 quick upvotes, mostly by readers who either a) were also victims of the misconceptions or b) figured, "hey that sounds reasonable". Then I posted a rather detailed, correct answer. The old answer was unaccepted, received a few downvotes, and my answer received ~200 upvotes over the next few days. This wasn't because of vote inflation. This was because a "decent looking" but incorrect answer was posted, and then a new answer changed the landscape. Don't mistake these kinds of things for the effect you describe; questions and answers can be a very dynamic thing.


As for the motivation, you've identified a theoretical effect (vote inflation) but have not stated why it is an issue.

Recall that the purpose of these sites is to provide good information. The fact is, regardless of the vote counts, in both of your examples the most useful answers are presented first, and sorted roughly in decreasing order of usefulness. So even if bandwagoning and vote inflation is happening, the information quality on the site is not diminished. There is no issue here.

Vote "inflation", in a manner of speaking, can happen on certain questions. For example, if the highest voted answer perfectly answers the question, users may not have need to scroll to the second answer. You see that on questions like What is the correct JSON content type?, but at the end of the day this doesn't matter at all. The first answer answers the question. The site serves its purpose and readers win. (Incidentally, if you sort the answers on that JSON question by time, then look at the vote counts, it does serve as a partial counter-example to the vote inflation claim).

Hiding votes hides quality-of-post information but with no real benefit, as no problems with information quality really exist as a result of visible vote counts.


As for some of your comments, just repeating them here:

I don't think bandwagoning happens as much as you think on these sites, but that is just my impression based on experience.

You write:

how many people are going to try a -8 voted answer

You're forgetting a massive point here: You're forgetting that if that answer is actually correct, that at least 8 or more of the possibly hundreds of thousands of involved and knowledgeable users on the sites who may see that answer and review (but not need to try) it upvote it, etc.

You also write:

How many people going to scroll past a +100 answer to find a better one?

Yeah, people will scroll past it if it doesn't answer the question. They have no reason to scroll past it if it's the correct answer. No issue with information quality there.

Don't forget: Far more people vote on questions and answers than just people who happened to find the question in a google search because they were looking for an answer. The active members of this community review the existing questions and answers. That's the strength of these sites in general.

As for what you deserve wrt privileges, we already know the reputation system isn't the most accurate way to map to privileges, if that's what you're aiming for, but it's decent enough.

There's one more big effect that you are not acknowledging: While the votes you garner from a single (or small handful of questions) may not accurately reflect your "community standing", over time, as you participate in many questions and answers, this evens out. (E.g. think of your English example as a way to balance out the other good answers you provided on that site that didn't get the votes you feel you deserved.) Do not make the mistake of placing too much importance on a small amount of samples in a high-noise sample set. While you may have a few questions here and there that "do not make sense" as far as votes go, the bigger picture is that if you've contributed dozens or hundreds of good quality posts, your community standing is reflected more accurately in that.

  • 1
    My argument about the C++ question makes more sense with the clarification that it is the duplicate question that has the equivalent answer. I will have to think about the rest of what you say. – imallett May 16 '15 at 18:31
  • @imallett I've updated. In fact, it is an even stronger counter example of the effect you describe now. Not only did the new, better answer exceed, in one day, the vote count of the five-year-old answer to the original old question, the old question was marked as a duplicate of the new one. This is almost the polar opposite of vote inflation, although it is across multiple questions instead of within the context of one. The old answer that was identical to yours was not the greatest, and having gained only 43 upvotes after 5 years and 130k+ views is indicative of its average quality. – Jason C May 16 '15 at 19:01
  • I.e. it may have taken an unfortunately long time, but the overall information quality on the has site finally stabilized wrt that topic. I like to imagine that had thelink2012 (answerer of current question) seen the original one when it was posted 5 years ago, they would have posted an answer that would now have a vote count in the hundreds or thousands, rather than in the 40's (but of course that is conjecture). (Incidentally, I've flagged the old question to request a full merge with the new one.) – Jason C May 16 '15 at 19:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .