Is the voting and reputation system sustainable? How can we improve it or maybe it should be replaced?

Question

Is voting still sustainable?

• The number of votes going down and the dynamics changing towards questions and answers only being relevant during the first few days.

• This makes the rating system, by means of a system that measures popularity (counts votes/likes), not very useful. Or at least, there is no nuanced differentiation and the voting is only useful for the purpose of extreme cases like separating posts that should be closed/deleted.

• Can we do something about this (like making a new ranking/voting/rating system; or motivate to make more votes), or are we doomed?

---

Voting is important because:

Voting is central to our model of providing quality questions and answers; it is how …

• ...good content rises to the top
• ...incorrect content falls to the bottom
• ...users who consistently provide useful content accrue reputation and are granted more privileges on the site

---

However we may wonder whether this system still works well. There are several reasons to believe that it is not so anymore (or without further actions it will be worse in the future).

• We can address some of this to more people gamifying the system, or more lower quality contributions. (rather than something intrinsic to the system)
• But also... the mere growth in number of questions makes the system unwieldy and difficult to navigate. The data below suggest that SE/SO seems to be growing too big to be handled/managed by simple up and down votes (votes with small frequency, large randomness/variance, and bias).

---

Note: the text below here discusses a bunch of data and graphs, which can be skipped, but for those who like to be involved in the discussion I highly suggest to take note of this as these graphs and data sketch an evolution that is going on with SE/SO and it shows/sketches how the voting is changing while SE/SO ages (and grows).

Low voting activity

The relative rate of voting is very low (questions and answers are getting lower scores)

See in the image below the development of questions by how many votes they receive as function in the first 30 days.

(From this query)

In this image a differentiation is made for classes. It shows how many questions are created each week, differentiated by the score they will obtain after 30 days. The curves show the changes in the growth of number of questions (of particular type).

What we can see is that the number of questions asked that acquire high scores in the first 30 days is decreasing and it is mostly questions with score 0 or 1 that are currently being made (this can be seen as degrading quality but it is also due to the scaling of the voting system which will be argued furher below.).

Very few questions receive a score different from zero, so the scoring system is not differentiating very well the best questions.

See also from the image in this query

The popularity type of voting system might differentiate questions and answers when there are a lot of votes (such that variance/probability evens out and becomes of less influence).

However, as the website SO/SE grow larger, the relative number of votes, the probability that a question gets a vote, is very low. This means that most questions (and the same is true for answers) do not get differentiated. The difference between 0, 1, or 2 votes (which are roughly ~50%, ~25% and ~12.5% of all the questions) is much influenced by chance, it depends on how many and which contributors pass by the question and whether or not a contributor thought about up/down-voting the question. It is only 12.5% of the remaining questions that really get attention of voters and become differentiated in quality (when getting more than two votes the randomness starts to be with less variance).

The absolute rate of voting is very low

See in the image below the development of the total number of votes (per month).

(From this query)

In this image the total number of votes that are being made in a particular month (the voting activity) is being split up in votes that are made on posts that were less than 1 week old and votes that are being made on posts that are more than 1 week old.

We can see that questions and answers that are one week or more old are nowadays hardly obtaining any votes (probably much related to not being being on the hot topics or active topics list). This is not just the case for individual questions receiving relatively less votes, but the total amount of votes on all questions together is decreasing.

The activity of Stack Exchange (at least the voting activity) has become mostly the activity of making questions and answering them. Of course, that is an important activity, but the idea of a database of questions and answers (where questions are being reused and helpful to others) is getting less strong. With this fast pace and only recent questions having activity, the platform runs the risk to turn into a helpdesk (for quick and dirty answers) rather than a knowledge base (for high quality information).

Inconsistent voting as function of time

The voting system is not a rating system. Basically people give -1 or +1 but they do not give a more nuanced image (like in rating of movies at IMDb which uses a ten point scale).

As a result the voting system on SE/SO is more like a popularity number. It gives an idea how many people have voted on the question. Any more nuanced image, like an average or other comparison of +1 and -1 scores is only useful for the very controversial questions and answers (it works more like close votes or delete votes, but not as a nuanced ranking of the quality).

So, beyond the quality of a question, it is very important as well how popular a question is. This can be seen based on several measures.

The effect of delay before a question is answered.

Answers that occur in the first few minutes/hours receive (a lot) higher scores on average. The image below shows this effect.

(From this query)

On the x-axis is the delay of an answer. On the y-axis is the mean score of answers with that delay. You can see that in the first day the average score drops for every hour later that the answer is posted.

This may (possibly) be attributed to the effect that early answers will become more popular, not because of the quality of the content but because the better exposure.

Eventually, at much later delay times (almost half a year) the average score of answers is high again. This might relate to 'true' quality of answers.

The effect of delay and timing of the question can be demonstrated more dramatically by plotting the score as a function of the time (hour of the day) that the question and answer had been posted. When this is done for a language-specific StackOverflow then we not only see that answers score better when they are posted quickly after the question, but also that the answers score best when they are posted from 10h to 16h.

(with this query for Spanish)

The age of questions and answers 1

The image below shows the development of average question and answer score in time.

(From this query)

We can see the average score of questions and answers decreases with the time. Newer posts will have lower scores. This is not only because older posts had more time to acquire votes, it is also because the score acquired within the first 30 days is decreasing for both questions and answers.

The age of questions and answers 2

Thus the voting system, as the site is getting older, is placing relatively much less votes on newer questions.

The votes are getting more and more diluted among a larger bulk of questions and answers. And as a result the scores on new posts start to become less and less meaningful. (this closes the circle a bit to the first point, the relative/absolute rate of voting is low)

Nowadays questions and answers only acquire votes in the first few days. It takes more than a year before a question/answer grows in score as what it did in the first month (in terms of the average score for all questions and answers).

In the early years of SE/SO a question/answer would acquire (on average) almost as many votes in later months as in the first months (questions even get relatively more votes as they age).

This is shown in the image below which shows the distribution of score as function of age.

(Using this query)

You can see that mostly in the first month questions obtain increase in their (average) score.

There is a large discrepancy between the questions from 10 years ago and questions now. The newer questions have much smaller average monthly increase of score, in comparison to the older questions. Also, the older questions still had some score increase after the first month (and the rate of increase is even rising as the question ages, possibly due to additional answers and growing activity). For the newer questions there is (on average) much less going on after one month.

---

One could play around a bit with this query which tracks the score in time for different answers to a particular question. There is a large variation how questions accumulate their scores in time. Many questions acquire votes during short periods (presumably when they occur on the front page), some others also grow more continuously (due to visitors stumbling on them).

See for instance the development of score for this question on Cross Validated.

The change of score occurs in several discrete steps. This is obviously related to the question being bumped up whenever a new answer appears or whenever an answer is changed.

It is difficult to say whether the top answer grows because

• it is simply the best (it is not strange that the top answer grows faster; but there are indications that it is not necessary too grow because of being the best. gbjbaanb describes in his answer an old question with large growth of score for an answer which describes a bad practice in programming, and that is not an unique case)
• or because being the top answer creates a reinforcing loop making it difficult to become overtaken by other answers?

But what is clear is that a question/answer that gets bumped up several times by edits or additions is more likely to pick up extra votes. (this is an idea to make another graph describing clustering of votes; the ratio of the number of votes and the number of days that has been voted; a high ratio indicates that votes are mostly occurring together and indicates voting because of homepage visibility; a bit like separating homepage visits into different days or in unique visitors)

Answer 1

The gamification has led to some detrimental effects:

• The first answers always receive more votes than later answers, rewarding quick, sloppy answering.
• The site has become addictive for many users who answer questions they are not truly qualified to answer, because even one upvote will still get you more reputation than four downvotes.
• Voters are not necessarily qualified to distinguish good from bad answers. They therefore tend to vote for what they believe is correct.

See also:

Writing SE answer: How do you write a Stack Exchange answer?

Answer 2

Disclaimer: There was a time when I was the most active voter on all of SE and the first person to vote on questions and answers 50,000 100,000 times, so I tend to see "more voting" as the cure-all for a lot of SE's problems.

I've always felt that voting is one of the two key factors in making Stack Exchange sites useful (the other one being the strict focus on Q&A, not discussion). As the OP shows, the amount of voting on an SE site tends to fall off over time, and in some cases it drops off dramatically. Server Fault is 10 years old and lack of voting there was noted as a problem 7 years ago.

So the answer to the question in your title is that yes, apparently voting is not sustainable. But when you look at the more detailed questions in your post, I think it's less straightforward:

This makes the rating system, by means of a system that measures popularity (counts votes/likes), not very useful. Or at least, there is no nuanced differentiation and the voting is only useful for the purpose of extreme cases like separating posts that should be closed/deleted.

Voting was never about popularity, it was supposed to reflect quality. If lots of people are voting up or down based on whether they think it's a good or bad post, then good questions or answers will (usually) get higher scores than bad ones.

You're right, there's not much nuance to the voting on Server Fault: There are typically 100-200 questions per day, only about 10% get any answer, and most posts get at most a couple votes. But a few posts get more votes (typically both the Q and the few answers) so they stand out a bit from all the "meh" ones.

The number of questions has - of course - a big impact on your first point:

The number of votes going down and the dynamics changing towards questions and answers only being relevant during the first few days.

On Server Fault I'd say it's less than a few days, and on SO with even more questions it's probably more like a few hours at most. OTOH, on "slower" sites, there seem to be more answers per question.

Can we do something about this (like making a new ranking/voting/rating system; or motivate to make more votes), or are we doomed?

Voting is already easy and simple, but on every SE site where I'm active, I've seen meta posts asking "why don't people vote more." I've never seen anything done or suggested to increase voting, so I think any future plans for SE have to include the fact that voting will always be low.

I think low levels of voting dooms the idea of SE sites functioning as a library of ranked, high-quality Q&A in the long term. I've concluded that there's a sort of demographic shift that takes place on most SE sites from the library model to whatever you would call Server Fault and other older, high-volume sites, but it's up to SE Inc. to figure out what the long-term model is and what to do with it.

Answer 3

It seems to me that any system of that kind will become less efficient over the time. A somewhat similar story is the one of citations in academia. Originally they were counted in order to "measure" the scientific impact of a publication. The more other works refer to a given publication, the more significant the result.

However, over the time such systems tend to get more and more gamed. In academia, there are clubs of authors who frenetically cite the works of other members of the club. Even though voting is private, I believe to have soptted similar clubs on the SE sites.

Other manifestations of such gaming practices are nicely surveyed in I am not the way you speak's answer.

It seems to be a general rule that whenever you try to introduce a metric that is supposed to measure the quality of something, there will be ways of getting a rather high score on that metric without really providing the quality. So you may come up with a new metric, which may work fine for some amount of time, but you can be (more or less) sure that some will find ways to manipulate it.

This suggests that the answer to the question

Is the voting and reputation system sustainable?

is

It will continue losing its usefulness.

I am also afraid that the "How can we improve it" part in

How can we improve it or maybe it should be replaced?

will not have a once-and-for-all answer.

One may think about several improvements. (I apologize in advance if these proposals exist somewhere already, and will be happy to add appropriate citations. I am not claiming I invented these proposals.)

You seem to be worried about low-quality answers, which get many votes and sometimes even accepted, and so am I. In academia there is a refereeing system, which originally was introduced to weed out low quality papers. Even though this refereeing system is not always fair, it does help rejecting contributions of very low quality. So one might think of installing a couple of referees on each site who check the correctness of answers. This proposal goes in the direction of Maximus Minimus' answer with the difference that referees are supposed to have more tools to delete/reject posts than ordinary high-reputation users. These referees will be complementary to the moderators, whose main concern is to impose a filter on how users say something, but not so much on the correctness of the content. Whether or not this is feasible, i.e. whether one will really find enough qualified users who volunteer to referee, and whether this will lead to even more sociological issues, I do not know. But you were asking for a suggestion, and this is one. If it works, it will at least install a filter which spares others from getting answers that are plainly wrong. As mentioned above, this may potentially help improving the overall quality of posts for some time, but very likely will lose its usefulness once it is widely known how to game this system.

Answer 4

There will probably no single answer that covers all aspects that have to be considered here. But some thoughts:

Interestingly, the title asked about voting and reputation. But in the question itself, you did not mention reputation at all. I think that there is not much doubt that voting per se is useful. So I think that the question is really whether voting still serves the purpose that it should serve (considering the voting changes that you observed), and not so much about its connection to reputation.

^{(But regarding the latter: I think that when there is a voting system (and answers can be attributed to individuals), then the votes will also always be an implicit measure of reputation - regardless of whether it is "the number of upvotes" or "the number of upvotes times ten")}

---

Low voting activity

It's hard to draw conclusions form the comparison Stack Overflow to Cross Validated - or to any other network site, for that matter: I'm pretty sure that many low-quality questions are asked on Stack Overflow that could justifiably be asked on a different site. The flippant way of saying this: On Stack Overflow, people might just throw their random computer problem on the site that solved all the random computer problems that they had before. If someone knew that a question was well-suited for Cross Validated, then it would be more likely that it was a good question...

I also think that the differences in the graphs can largely be attributed to the diversity of Stack Overflow. (No, I'm not referring to the "diversity of users in terms of personal attributes that are totally irrelevant here", but to the the diversity of topics). People who are active on Cross Validated are far more likely to be specialized in the (comparatively narrow) fields of the site. But everybody is on Stack Overflow, and the range of topics is far too broad

(There are ~1700 Tags on Cross Validated, but nearly 60000 (!) tags on Stack Overflow. Let's assume that people are experts (and voting) in less than 1% of the tags that a site is about...)

---

The absolute rate of voting is very low

This is related to the low voting activity in some sense: The influx of questions on stack overflow is too high. Cross Validated is also one of the larger sites, but still, a question may be on the front page after an hour. On Stack Overflow, the time for a question to remain on the front page may be less than 5 minutes.

(The fact that the voting activity basically drops to zero after a week is nevertheless concerning)

---

Inconsistent voting as function of time

I don't think that a more "nuanced" voting mechanism (beyond +1/-1) could help here, unless there was an accompanying update of how the votes affect the visibility of a question.

Right now, the popularity plays a role in two regards: When I scroll over the main page, and see a question with a title like "What causes this error in JavaScript?", I'll definitely not read it... unless it has 10 upvotes, and my geek-curiosity is triggered. The other aspect is the "10k mod tools". When a question has enough votes to appear in the "Questions with extreme votes" list, it's bound to attract further votes (usually, in the direction that caused it to appear in the list in the first place).

(Beyond that: I had a look at the queries here, and am not sure whether the times that you mentioned are right. Specifically, I wonder about the scale of the x-axis and how you came up with "half a year", considering that the query with DATEDIFF(hh, ...) seems to be about hours, but I'm not an expert here. Also, some of the queries don't directly generate the graphs as they are shown in the question, so I hesitate to say something about this)

---

How can we improve it?

The ways in which the voting system on Stack Overflow in particular is distorted are, (subjectively, but roughly aligned to your observations) :

• There are too many different topics on Stack Overflow
• There are too many questions asked and answered too quickly on Stack Overflow (and some aspects that you mentioned seem to be related to the "Fastest Gun In The West" problem)
• Recently, people talked about observing "Pity Upvotes", where things have been upvoted to counter downvotes...
• A severe problem: My impression is that many people are upvoting crappy questions because they can (quickly) answer them. Bad questions and bad answers are then "validating" each other - basically as a "temporary, two-person voting ring" - but nobody except for the asker and the answerer would vote for that later.

We, individually, cannot improve that - only by voting consistently. Further measures (for example, higher standards (as in "more elitism")) are beyond our control.

Answer 5

I know I'm out of my depth here, I tried to decipher the graphs but I found them confusing.

All I know is, when a site such as Stack Overflow, attracts millions of visitors daily and the number of users between 1 and 50 rep swamp the site with boring repetitive Googleable questions, and users who are only interested in badges and points post "cheap" easy answers, the quality goes down the toilet.

The users who have the rep to vote, will continue to do so as long as they enjoy the “game”. Users who care about quality, will vote less and less as their enjoyment of the site sinks lower and lower.

The solution? Drastically, radically, mercilessly, reduce the number of questions from new users flooding SO.

Give new SO users a quota of five questions to answer. If none of these answers attract at least a positive score (+1), those users are suspended from answering further questions. If they need to post questions because their livelihood depends on it, then charge them a fee: 10 dollars gives them the opportunity to post twenty questions. Limit the answers to hi-rep users (25k) to answer these questions and pay them in swag or something or another. Watch those answers and upvotes skyrocket.

I know this answer will be downvoted to hell.

Answer 6

I agree. Gaining reputation by answering questions is very often hard work. Nonetheless, I consider it essential for a large group of people:

• New users are looking for reputation to get to higher privileges (I started my whole Stack Overflow career, because I was behind my companies proxy, and thus, back then, getting to 125 or 250 or whatever significantly improved the "SO experience").
• Gamification users are looking for reputation, because that is their core motivation.
• Even top users with other priorities (like: "I am here to answer questions by readers of my books") don't shy away from collecting reputation.

So, please: as a gamification person I didn't like the question-weight update, and any further ideas to significantly change the reputation/voting system doesn't sound attractive at all.

And seriously: you don't want to annoy those people that contribute the majority of the high-quality content.

Answer 7

It seems our current voting system is like democracy; it's not the best system we can create, but all others are worse.

Now, it does not mean that we cannot improve the current system. I, for one, do not like the valuation increase of points for a question. Questions are easy, good questions are harder, and good answers are hardest. The previous system handled that better than the current.

What we might need is a way to devaluate older answers and questions. Good and old answers are like a nice bank account with a good return rate: you can still get points from it without doing any work. But not sure on how to make that work.

In the end, you want to reward the current answer givers and honor the previous ones.

Answer 8

The voting system can be sustainable provided SE is able to attract and retain knowledgeable subject matter experts. A loose working definition of such a person might be someone who is capable of hitting 20k on a medium volume site.

A voting system built around usefulness and quality of questions and answers absolutely requires such people participating, otherwise the general trend is downwards, in both quality and quantity, and SE will eventually become what its founders set it up to oppose.

The voting system doesn't need to change; SE does.

Answer 9

The idea is that voting means the best answers get promoted, and the reputation system attracts experts. But experts do not affect the voting.

Check this comment to an answer - 18 upvotes to it, yet 786 upvotes for the truly bad answer, and it is still accepted as correct. (It's far from the only one, but it is possibly the one that is provably wrong due to Eric's involvement).

Look at Eric's answers, 124 pages of answers, only 4 are in the hundreds. Voting is therefore all about popularity and herd instinct, not quality. People will uptick answers that look good, or already have lots of votes. They will also downvote similarly.

The answer, perhaps, is to make voting harder. To vote on something you must also do something to make your vote worthwhile, comment or something to admit you think it's a good answer, and thus stop the mindless upticking because it already has lots of upticks. I doubt such a practice would be workable or preferred however.

Answer 10

Allow an explicit blank vote as well

This change to the voting mechanism which will:

• Not change the game-play of the gamification that voting brings (reputation will still be based on the +1 and -1 votes).

• Yet, it will be able to provide extra information and make the votes more meaningful in differentiating quality of questions.

By allowing to vote blank (explicitly) a contributor can tell that they have read an answer/question and that they did not care about up- or down-voting. Normally we read a question/answer and either vote or not. But when we do not vote, then the voting-system does not know that (The system doesn't know whether contributors have made the consideration of voting or whether they just not yet read the questions/answers).

With the information of blank votes an algorithm can verify whether or not the questions and answers have been equally read (and possibly create some additional - more nuanced - score value aside from the current one). It can do this especially well when contributors are motivated to cast votes on all answers to a question (because then a repeated measures comparison can be made) and when they are notified of questions where their votes are incomplete (such that users can re-evaluate when new questions are added).

The advantage of this system is that it is only a small adaptation and does not really influence the up/down +/- 1 scoring and related reputation system. Already now people are implicitly voting blank. The difference is that now this can be done explicitly and the system might try to stimulate voting.

The disadvantage of this system is that, as simple as it seems, it may not be so easy technologically and requires some changes to both front and backend of the website. Also, whatever the special algorithm is going to do, it might be somewhat opaque how the score is created (although it does not need to be difficult).

---

Change the expressions of the score on questions and answers, by some transformed number.

This may have the psychological effect that large differences are being reduced. Users may need to be guided in understanding that a difference between score 0,1,2,3 is a much larger difference than the difference between score 10 and 100.

The effects is that the scores will be easier to compare, and in addition the contributors/voters might be having less tendencies to automatically vote on the top answers ("if 100 people or more before me found this a good question then I should find that as well")

---

Make voting more meaningful to a person, by linking to recommendations or other site functions.

SE/SO could place algorithms based on user votes that help to suggest new interesting questions and answers for users.

Then users might be more motivated to vote in order to get better recommendations. The advantage is that users will be considering their voting more carefully and it will more accurately resemble what users find good or not. The disadvantage is that users will be voting too much on their interests and not necessarily so much on the quality.

Answer 11

Allow No-Account visitors to somehow Vote or contribute Reputation

and / or

Gain Reputation based on Views

---

• Getting feedback from anonymous users
• Anonymous user feedback now in testing
• Use upvote button for "Thank You" if you don't have enough rep

---

The relative rate of voting is very low

The absolute rate of voting is very low

[...] the idea of a database of questions and answers (where questions are being reused and helpful to others) is getting less strong. With this fast pace only recent questions activity, the platform runs the risk to turn into a helpdesk (for quick and dirty answers) rather than a knowledge base (for high quality information).

---

I have several 75,000-view Questions on Stack Overflow. The Questions each have only ~100 points. The accepted answers have ~100 points. The remaining good answers attract a handful of upvotes, 5-30 points.

It seems clear that MANY people are being helped by those questions and answers. But it appears that only ~100 have been.

Likely lots of Google SERP-based visitors. (can we break down View stats of by account vs. no-account?)

Find a way to reward highly-viewed search result Questions / Answers that does not involve upvotes from people who have an account and the sufficient privileges to vote.

Perhaps allow anonymous users to upvote, perhaps at a lower-per-vote-rep. Or begin rewarding reputation for Views.

Encourage no-account voters to upgrade to an account, using "don't lose all your anonymous votes/content!" as a better upsell than the existing workflow.

Something would have to be done to normalize HNQ traffic and other external high-traffic links.