Is voting still sustainable?
The number of votes going down and the dynamics changing towards questions and answers only being relevant during the first few days.
This makes the rating system, by means of a system that measures popularity (counts votes/likes), not very useful. Or at least, there is no nuanced differentiation and the voting is only useful for the purpose of extreme cases like separating posts that should be closed/deleted.
Can we do something about this (like making a new ranking/voting/rating system; or motivate to make more votes), or are we doomed?
Voting is important because:
Voting is central to our model of providing quality questions and answers; it is how …
- ...good content rises to the top
- ...incorrect content falls to the bottom
- ...users who consistently provide useful content accrue reputation and are granted more privileges on the site
However we may wonder whether this system still works well. There are several reasons to believe that it is not so anymore (or without further actions it will be worse in the future).
- We can address some of this to more people gamifying the system, or more lower quality contributions. (rather than something intrinsic to the system)
- But also... the mere growth in number of questions makes the system unwieldy and difficult to navigate. The data below suggest that SE/SO seems to be growing too big to be handled/managed by simple up and down votes (votes with small frequency, large randomness/variance, and bias).
Note: the text below here discusses a bunch of data and graphs, which can be skipped, but for those who like to be involved in the discussion I highly suggest to take note of this as these graphs and data sketch an evolution that is going on with SE/SO and it shows/sketches how the voting is changing while SE/SO ages (and grows).
Low voting activity
The relative rate of voting is very low (questions and answers are getting lower scores)
See in the image below the development of questions by how many votes they receive as function in the first 30 days.
In this image a differentiation is made for classes. It shows how many questions are created each week, differentiated by the score they will obtain after 30 days. The curves show the changes in the growth of number of questions (of particular type).
What we can see is that the number of questions asked that acquire high scores in the first 30 days is decreasing and it is mostly questions with score 0 or 1 that are currently being made (this can be seen as degrading quality but it is also due to the scaling of the voting system which will be argued furher below.).
Very few questions receive a score different from zero, so the scoring system is not differentiating very well the best questions.
See also from the image in this query
The popularity type of voting system might differentiate questions and answers when there are a lot of votes (such that variance/probability evens out and becomes of less influence).
However, as the website SO/SE grow larger, the relative number of votes, the probability that a question gets a vote, is very low. This means that most questions (and the same is true for answers) do not get differentiated. The difference between 0, 1, or 2 votes (which are roughly ~50%, ~25% and ~12.5% of all the questions) is much influenced by chance, it depends on how many and which contributors pass by the question and whether or not a contributor thought about up/down-voting the question. It is only 12.5% of the remaining questions that really get attention of voters and become differentiated in quality (when getting more than two votes the randomness starts to be with less variance).
The absolute rate of voting is very low
See in the image below the development of the total number of votes (per month).
In this image the total number of votes that are being made in a particular month (the voting activity) is being split up in votes that are made on posts that were less than 1 week old and votes that are being made on posts that are more than 1 week old.
We can see that questions and answers that are one week or more old are nowadays hardly obtaining any votes (probably much related to not being being on the hot topics or active topics list). This is not just the case for individual questions receiving relatively less votes, but the total amount of votes on all questions together is decreasing.
The activity of Stack Exchange (at least the voting activity) has become mostly the activity of making questions and answering them. Of course, that is an important activity, but the idea of a database of questions and answers (where questions are being reused and helpful to others) is getting less strong. With this fast pace and only recent questions having activity, the platform runs the risk to turn into a helpdesk (for quick and dirty answers) rather than a knowledge base (for high quality information).
Inconsistent voting as function of time
The voting system is not a rating system. Basically people give -1 or +1 but they do not give a more nuanced image (like in rating of movies at IMDb which uses a ten point scale).
As a result the voting system on SE/SO is more like a popularity number. It gives an idea how many people have voted on the question. Any more nuanced image, like an average or other comparison of +1 and -1 scores is only useful for the very controversial questions and answers (it works more like close votes or delete votes, but not as a nuanced ranking of the quality).
So, beyond the quality of a question, it is very important as well how popular a question is. This can be seen based on several measures.
The effect of delay before a question is answered.
Answers that occur in the first few minutes/hours receive (a lot) higher scores on average. The image below shows this effect.
On the x-axis is the delay of an answer. On the y-axis is the mean score of answers with that delay. You can see that in the first day the average score drops for every hour later that the answer is posted.
This may (possibly) be attributed to the effect that early answers will become more popular, not because of the quality of the content but because the better exposure.
Eventually, at much later delay times (almost half a year) the average score of answers is high again. This might relate to 'true' quality of answers.
The effect of delay and timing of the question can be demonstrated more dramatically by plotting the score as a function of the time (hour of the day) that the question and answer had been posted. When this is done for a language-specific StackOverflow then we not only see that answers score better when they are posted quickly after the question, but also that the answers score best when they are posted from 10h to 16h.
(with this query for Spanish)
The age of questions and answers 1
The image below shows the development of average question and answer score in time.
We can see the average score of questions and answers decreases with the time. Newer posts will have lower scores. This is not only because older posts had more time to acquire votes, it is also because the score acquired within the first 30 days is decreasing for both questions and answers.
The age of questions and answers 2
Thus the voting system, as the site is getting older, is placing relatively much less votes on newer questions.
The votes are getting more and more diluted among a larger bulk of questions and answers. And as a result the scores on new posts start to become less and less meaningful. (this closes the circle a bit to the first point, the relative/absolute rate of voting is low)
Nowadays questions and answers only acquire votes in the first few days. It takes more than a year before a question/answer grows in score as what it did in the first month (in terms of the average score for all questions and answers).
In the early years of SE/SO a question/answer would acquire (on average) almost as many votes in later months as in the first months (questions even get relatively more votes as they age).
This is shown in the image below which shows the distribution of score as function of age.
You can see that mostly in the first month questions obtain increase in their (average) score.
There is a large discrepancy between the questions from 10 years ago and questions now. The newer questions have much smaller average monthly increase of score, in comparison to the older questions. Also, the older questions still had some score increase after the first month (and the rate of increase is even rising as the question ages, possibly due to additional answers and growing activity). For the newer questions there is (on average) much less going on after one month.
One could play around a bit with this query which tracks the score in time for different answers to a particular question. There is a large variation how questions accumulate their scores in time. Many questions acquire votes during short periods (presumably when they occur on the front page), some others also grow more continuously (due to visitors stumbling on them).
See for instance the development of score for this question on Cross Validated.
The change of score occurs in several discrete steps. This is obviously related to the question being bumped up whenever a new answer appears or whenever an answer is changed.
It is difficult to say whether the top answer grows because
- it is simply the best (it is not strange that the top answer grows faster; but there are indications that it is not necessary too grow because of being the best. gbjbaanb describes in his answer an old question with large growth of score for an answer which describes a bad practice in programming, and that is not an unique case)
- or because being the top answer creates a reinforcing loop making it difficult to become overtaken by other answers?
But what is clear is that a question/answer that gets bumped up several times by edits or additions is more likely to pick up extra votes. (this is an idea to make another graph describing clustering of votes; the ratio of the number of votes and the number of days that has been voted; a high ratio indicates that votes are mostly occurring together and indicates voting because of homepage visibility; a bit like separating homepage visits into different days or in unique visitors)