39

Is voting still sustainable?

  • The number of votes going down and the dynamics changing towards questions and answers only being relevant during the first few days.

  • This makes the rating system, by means of a system that measures popularity (counts votes/likes), not very useful. Or at least, there is no nuanced differentiation and the voting is only useful for the purpose of extreme cases like separating posts that should be closed/deleted.

  • Can we do something about this (like making a new ranking/voting/rating system; or motivate to make more votes), or are we doomed?


Voting is important because:

Voting is central to our model of providing quality questions and answers; it is how …

  • ...good content rises to the top
  • ...incorrect content falls to the bottom
  • ...users who consistently provide useful content accrue reputation and are granted more privileges on the site

However we may wonder whether this system still works well. There are several reasons to believe that it is not so anymore (or without further actions it will be worse in the future).

  • We can address some of this to more people gamifying the system, or more lower quality contributions. (rather than something intrinsic to the system)
  • But also... the mere growth in number of questions makes the system unwieldy and difficult to navigate. The data below suggest that SE/SO seems to be growing too big to be handled/managed by simple up and down votes (votes with small frequency, large randomness/variance, and bias).

Note: the text below here discusses a bunch of data and graphs, which can be skipped, but for those who like to be involved in the discussion I highly suggest to take note of this as these graphs and data sketch an evolution that is going on with SE/SO and it shows/sketches how the voting is changing while SE/SO ages (and grows).

Low voting activity

The relative rate of voting is very low (questions and answers are getting lower scores)

See in the image below the development of questions by how many votes they receive as function in the first 30 days.

(From this query)

Decline in large number of votes

In this image a differentiation is made for classes. It shows how many questions are created each week, differentiated by the score they will obtain after 30 days. The curves show the changes in the growth of number of questions (of particular type).

What we can see is that the number of questions asked that acquire high scores in the first 30 days is decreasing and it is mostly questions with score 0 or 1 that are currently being made (this can be seen as degrading quality but it is also due to the scaling of the voting system which will be argued furher below.).

Very few questions receive a score different from zero, so the scoring system is not differentiating very well the best questions.

See also from the image in this query

Distribution of number of votes on questions

The popularity type of voting system might differentiate questions and answers when there are a lot of votes (such that variance/probability evens out and becomes of less influence).

However, as the website SO/SE grow larger, the relative number of votes, the probability that a question gets a vote, is very low. This means that most questions (and the same is true for answers) do not get differentiated. The difference between 0, 1, or 2 votes (which are roughly ~50%, ~25% and ~12.5% of all the questions) is much influenced by chance, it depends on how many and which contributors pass by the question and whether or not a contributor thought about up/down-voting the question. It is only 12.5% of the remaining questions that really get attention of voters and become differentiated in quality (when getting more than two votes the randomness starts to be with less variance).

The absolute rate of voting is very low

See in the image below the development of the total number of votes (per month).

(From this query)

decline in absolute number of votes

In this image the total number of votes that are being made in a particular month (the voting activity) is being split up in votes that are made on posts that were less than 1 week old and votes that are being made on posts that are more than 1 week old.

We can see that questions and answers that are one week or more old are nowadays hardly obtaining any votes (probably much related to not being being on the hot topics or active topics list). This is not just the case for individual questions receiving relatively less votes, but the total amount of votes on all questions together is decreasing.

The activity of Stack Exchange (at least the voting activity) has become mostly the activity of making questions and answering them. Of course, that is an important activity, but the idea of a database of questions and answers (where questions are being reused and helpful to others) is getting less strong. With this fast pace and only recent questions having activity, the platform runs the risk to turn into a helpdesk (for quick and dirty answers) rather than a knowledge base (for high quality information).

Inconsistent voting as function of time

The voting system is not a rating system. Basically people give -1 or +1 but they do not give a more nuanced image (like in rating of movies at IMDb which uses a ten point scale).

As a result the voting system on SE/SO is more like a popularity number. It gives an idea how many people have voted on the question. Any more nuanced image, like an average or other comparison of +1 and -1 scores is only useful for the very controversial questions and answers (it works more like close votes or delete votes, but not as a nuanced ranking of the quality).

So, beyond the quality of a question, it is very important as well how popular a question is. This can be seen based on several measures.

The effect of delay before a question is answered.

Answers that occur in the first few minutes/hours receive (a lot) higher scores on average. The image below shows this effect.

(From this query)

score as function of delay

On the x-axis is the delay of an answer. On the y-axis is the mean score of answers with that delay. You can see that in the first day the average score drops for every hour later that the answer is posted.

This may (possibly) be attributed to the effect that early answers will become more popular, not because of the quality of the content but because the better exposure.

Eventually, at much later delay times (almost half a year) the average score of answers is high again. This might relate to 'true' quality of answers.

The effect of delay and timing of the question can be demonstrated more dramatically by plotting the score as a function of the time (hour of the day) that the question and answer had been posted. When this is done for a language-specific StackOverflow then we not only see that answers score better when they are posted quickly after the question, but also that the answers score best when they are posted from 10h to 16h.

(with this query for Spanish)

the impact of question and answer posting time on answer score

The age of questions and answers 1

The image below shows the development of average question and answer score in time.

(From this query)

old posts with high scores

We can see the average score of questions and answers decreases with the time. Newer posts will have lower scores. This is not only because older posts had more time to acquire votes, it is also because the score acquired within the first 30 days is decreasing for both questions and answers.

The age of questions and answers 2

Thus the voting system, as the site is getting older, is placing relatively much less votes on newer questions.

The votes are getting more and more diluted among a larger bulk of questions and answers. And as a result the scores on new posts start to become less and less meaningful. (this closes the circle a bit to the first point, the relative/absolute rate of voting is low)

Nowadays questions and answers only acquire votes in the first few days. It takes more than a year before a question/answer grows in score as what it did in the first month (in terms of the average score for all questions and answers).

In the early years of SE/SO a question/answer would acquire (on average) almost as many votes in later months as in the first months (questions even get relatively more votes as they age).

This is shown in the image below which shows the distribution of score as function of age.

(Using this query)

score distribution over age of post

You can see that mostly in the first month questions obtain increase in their (average) score.

There is a large discrepancy between the questions from 10 years ago and questions now. The newer questions have much smaller average monthly increase of score, in comparison to the older questions. Also, the older questions still had some score increase after the first month (and the rate of increase is even rising as the question ages, possibly due to additional answers and growing activity). For the newer questions there is (on average) much less going on after one month.


One could play around a bit with this query which tracks the score in time for different answers to a particular question. There is a large variation how questions accumulate their scores in time. Many questions acquire votes during short periods (presumably when they occur on the front page), some others also grow more continuously (due to visitors stumbling on them).

See for instance the development of score for this question on Cross Validated.

growth of score for questions

The change of score occurs in several discrete steps. This is obviously related to the question being bumped up whenever a new answer appears or whenever an answer is changed.

It is difficult to say whether the top answer grows because

  • it is simply the best (it is not strange that the top answer grows faster; but there are indications that it is not necessary too grow because of being the best. gbjbaanb describes in his answer an old question with large growth of score for an answer which describes a bad practice in programming, and that is not an unique case)
  • or because being the top answer creates a reinforcing loop making it difficult to become overtaken by other answers?

But what is clear is that a question/answer that gets bumped up several times by edits or additions is more likely to pick up extra votes. (this is an idea to make another graph describing clustering of votes; the ratio of the number of votes and the number of days that has been voted; a high ratio indicates that votes are mostly occurring together and indicates voting because of homepage visibility; a bit like separating homepage visits into different days or in unique visitors)

  • Comments are not for extended discussion; this conversation has been moved to chat. They're relevant for most part so...better they get archived than nuked. – Journeyman Geek Nov 22 '19 at 3:26

11 Answers 11

41

The gamification has led to some detrimental effects:

  • The first answers always receive more votes than later answers, rewarding quick, sloppy answering.
  • The site has become addictive for many users who answer questions they are not truly qualified to answer, because even one upvote will still get you more reputation than four downvotes.
  • Voters are not necessarily qualified to distinguish good from bad answers. They therefore tend to vote for what they believe is correct.

See also:

Writing SE answer: How do you write a Stack Exchange answer?

  • indeed, there are many cases on SO where answers attract upvotes, but if you were to actually run the code, you'd see that it either fails or produces something other than the desired outcome. It's surely too expensive of a feature, but wouldn't it be nice if there was an interpreter or compiler that would actually check to see if a code snippet throws an error before allowing an answer post? – De Novo Nov 19 '19 at 18:01
  • 1
    You're missing the point of longer terms for questions and answers, and how they tend to become useful for future research of other users. – πάντα ῥεῖ Nov 19 '19 at 18:11
  • All true, and one could list many, many more - but that's not really much for an "answer to this question" :-/ (I.e. no downvote, but no uvpote either...) – Marco13 Nov 19 '19 at 18:19
  • 7
    SE should randomize the order of answers and hide their scores for the first day or so to give everyone a chance to answer, instead of all the upvotes going to the first one in the list. – SurpriseDog Nov 19 '19 at 19:26
  • 2
    @SurpriseDog Those are good thoughts, but it doesn't change the fact that most who visit a question and vote on its answers will not return to read later-coming answers, so the first answers get more views and have a higher likelihood of getting votes. To change that, answer would have to be held for a while and then all published at the same time, which will make asking unattractive (and cause lots of duplicate answers). – I am not the way you speak Nov 19 '19 at 19:31
  • 1
    @Iamnotthewayyouspeak There are ways to fairly deal with items that get different amounts of exposure. – Stop Harming the Community Nov 19 '19 at 22:00
  • 3
    Many accepted, top-voted answers that are just plain wrong or just really bad or even so bad the language author commented on it. – gbjbaanb Nov 19 '19 at 23:53
  • @rockwalrus-stopharmingMonica Maybe the site I main at is special but that wouldn't help much. The not-terrible answers normally don't get downvotes. – Peilonrayz Nov 20 '19 at 0:01
  • @Peilonrayz the author suggests dividing by the number of times the item has been viewed in a case like that. – Stop Harming the Community Nov 20 '19 at 1:28
  • @rockwalrus-stopharmingMonica So first -> more views, more views -> top, top -> more views? – Peilonrayz Nov 20 '19 at 1:36
  • @Peilonrayz Dividing by the number of views tends to push content that doesn't get voted on towards the center of the list. It has to be upvoted continually to stay on top. – Stop Harming the Community Nov 20 '19 at 15:01
  • @rockwalrus-stopharmingMonica not a bad idea in general, but you'd need a "views by answer" estimate that can clearly discriminate between answers on the same page. This would be a new feature and has been avoided in the past. – De Novo Nov 21 '19 at 16:07
  • The first bullet point is the fastest gun in the west phenomenon (FGITW). – P.Mort. - forgot Clay Shirky_q Nov 21 '19 at 16:48
16

Disclaimer: There was a time when I was the most active voter on all of SE and the first person to vote on questions and answers 50,000 100,000 times, so I tend to see "more voting" as the cure-all for a lot of SE's problems.

I've always felt that voting is one of the two key factors in making Stack Exchange sites useful (the other one being the strict focus on Q&A, not discussion). As the OP shows, the amount of voting on an SE site tends to fall off over time, and in some cases it drops off dramatically. Server Fault is 10 years old and lack of voting there was noted as a problem 7 years ago.

So the answer to the question in your title is that yes, apparently voting is not sustainable. But when you look at the more detailed questions in your post, I think it's less straightforward:

This makes the rating system, by means of a system that measures popularity (counts votes/likes), not very useful. Or at least, there is no nuanced differentiation and the voting is only useful for the purpose of extreme cases like separating posts that should be closed/deleted.

Voting was never about popularity, it was supposed to reflect quality. If lots of people are voting up or down based on whether they think it's a good or bad post, then good questions or answers will (usually) get higher scores than bad ones.

You're right, there's not much nuance to the voting on Server Fault: There are typically 100-200 questions per day, only about 10% get any answer, and most posts get at most a couple votes. But a few posts get more votes (typically both the Q and the few answers) so they stand out a bit from all the "meh" ones.

The number of questions has - of course - a big impact on your first point:

The number of votes going down and the dynamics changing towards questions and answers only being relevant during the first few days.

On Server Fault I'd say it's less than a few days, and on SO with even more questions it's probably more like a few hours at most. OTOH, on "slower" sites, there seem to be more answers per question.

Can we do something about this (like making a new ranking/voting/rating system; or motivate to make more votes), or are we doomed?

Voting is already easy and simple, but on every SE site where I'm active, I've seen meta posts asking "why don't people vote more." I've never seen anything done or suggested to increase voting, so I think any future plans for SE have to include the fact that voting will always be low.

I think low levels of voting dooms the idea of SE sites functioning as a library of ranked, high-quality Q&A in the long term. I've concluded that there's a sort of demographic shift that takes place on most SE sites from the library model to whatever you would call Server Fault and other older, high-volume sites, but it's up to SE Inc. to figure out what the long-term model is and what to do with it.

6

It seems to me that any system of that kind will become less efficient over the time. A somewhat similar story is the one of citations in academia. Originally they were counted in order to "measure" the scientific impact of a publication. The more other works refer to a given publication, the more significant the result.

However, over the time such systems tend to get more and more gamed. In academia, there are clubs of authors who frenetically cite the works of other members of the club. Even though voting is private, I believe to have soptted similar clubs on the SE sites.

Other manifestations of such gaming practices are nicely surveyed in I am not the way you speak's answer.

It seems to be a general rule that whenever you try to introduce a metric that is supposed to measure the quality of something, there will be ways of getting a rather high score on that metric without really providing the quality. So you may come up with a new metric, which may work fine for some amount of time, but you can be (more or less) sure that some will find ways to manipulate it.

This suggests that the answer to the question

Is the voting and reputation system sustainable?

is

It will continue losing its usefulness.

I am also afraid that the "How can we improve it" part in

How can we improve it or maybe it should be replaced?

will not have a once-and-for-all answer.

One may think about several improvements. (I apologize in advance if these proposals exist somewhere already, and will be happy to add appropriate citations. I am not claiming I invented these proposals.)

You seem to be worried about low-quality answers, which get many votes and sometimes even accepted, and so am I. In academia there is a refereeing system, which originally was introduced to weed out low quality papers. Even though this refereeing system is not always fair, it does help rejecting contributions of very low quality. So one might think of installing a couple of referees on each site who check the correctness of answers. This proposal goes in the direction of Maximus Minimus' answer with the difference that referees are supposed to have more tools to delete/reject posts than ordinary high-reputation users. These referees will be complementary to the moderators, whose main concern is to impose a filter on how users say something, but not so much on the correctness of the content. Whether or not this is feasible, i.e. whether one will really find enough qualified users who volunteer to referee, and whether this will lead to even more sociological issues, I do not know. But you were asking for a suggestion, and this is one. If it works, it will at least install a filter which spares others from getting answers that are plainly wrong. As mentioned above, this may potentially help improving the overall quality of posts for some time, but very likely will lose its usefulness once it is widely known how to game this system.

  • 1
    You make an interesting comment that gamification is eventually braking down any voting system. However, I believe that gamification is not the worst reason for the system to break down. It is the growth of the site, with more and more questions and answers that makes that the presence of the newer posts is very much diluted. This is an effect independent from gamification. Therefore I believe that your suggestion may work in making sure that questions and answers receive equal attention. – Sextus Empiricus Nov 19 '19 at 20:45
  • @SextusEmpiricus Yes, these are certainly very valid considerations. It might be generally true that the larger a field grows, the worse the sociological issues become, so yes, the growth is a huge factor. (One may also think that the simplest solution could be to remove the incentive of gaming the system. But I am afraid that this will also decrease the incentive to participate quite a bit.) – Schrödinger's cat Nov 19 '19 at 21:16
  • 1
    Counterexample: the Advogato trust metric is a quality indicator that is provably ungamable except through social engineering. It's fairly different from SE rep because it only gives a true/false answer instead of a quantitative one. – Stop Harming the Community Nov 19 '19 at 21:36
  • @rockwalrus-stopharmingMonica I do not know anything about that metrics but as you say it does not seem to allow one to "rank" posts relative to each other. Wikipedia also mentions that "Despite the trust metric, posting privileges to the front page of Advogato have been gained by controversial individuals, leading some to claim Advogato's trust metric solution is faulty.", but as I said I don't know anything about this metric, let alone have the ability verify or falsify this statement. – Schrödinger's cat Nov 19 '19 at 21:41
  • 1
    @Schrödinger'scat The proof divides the community into three parts: good users, bad users, and confused users. Good users only trust good and confused users, while the other two groups can trust anyone. (The assumption behind the names is that bad users abuse however they want to while confused users just have misjudged the character of some of the bad users, but the proof doesn't assume that behavior.) The proof shows that the number of bad users that end up trusted can never be more than the number of confused users times a tunable constant. – Stop Harming the Community Nov 19 '19 at 21:52
  • @rockwalrus-stopharmingMonica As I said, I have no idea about this metric. But I fail to see how any metric can be immune against a large number of users "conspiring" by deciding to vote favorably on each others posts regardless of the content. – Schrödinger's cat Nov 19 '19 at 23:34
  • @Schrödinger'scat The algorithm starts by trusting a few hard-coded users, and then trust flows through them at a limited rate to the users they trust, and through those users at a limited rate, etc., with each user consuming one unit of trust until there is no more trust to flow. It gets a little more complicated to deal with users who trust each other, but those details aren't important. Users that aren't connected to the trust source never get any trust, while the number of users that can receive trust through a confused user is limited by the amount of trust that is allowed to flow through – Stop Harming the Community Nov 20 '19 at 1:21
  • 2
    @rockwalrus-stopharmingMonica Replace "few hard-coded users" by "few competent and fair referees" and we get rather close. ;-) That is, if one could make sure that there is a group of neutral and competent referees, a lot of problems could be solved. If it were just not for the if ... ;-) – Schrödinger's cat Nov 20 '19 at 1:24
5

There will probably no single answer that covers all aspects that have to be considered here. But some thoughts:

Interestingly, the title asked about voting and reputation. But in the question itself, you did not mention reputation at all. I think that there is not much doubt that voting per se is useful. So I think that the question is really whether voting still serves the purpose that it should serve (considering the voting changes that you observed), and not so much about its connection to reputation.

(But regarding the latter: I think that when there is a voting system (and answers can be attributed to individuals), then the votes will also always be an implicit measure of reputation - regardless of whether it is "the number of upvotes" or "the number of upvotes times ten")


Low voting activity

It's hard to draw conclusions form the comparison Stack Overflow to Cross Validated - or to any other network site, for that matter: I'm pretty sure that many low-quality questions are asked on Stack Overflow that could justifiably be asked on a different site. The flippant way of saying this: On Stack Overflow, people might just throw their random computer problem on the site that solved all the random computer problems that they had before. If someone knew that a question was well-suited for Cross Validated, then it would be more likely that it was a good question...

I also think that the differences in the graphs can largely be attributed to the diversity of Stack Overflow. (No, I'm not referring to the "diversity of users in terms of personal attributes that are totally irrelevant here", but to the the diversity of topics). People who are active on Cross Validated are far more likely to be specialized in the (comparatively narrow) fields of the site. But everybody is on Stack Overflow, and the range of topics is far too broad

(There are ~1700 Tags on Cross Validated, but nearly 60000 (!) tags on Stack Overflow. Let's assume that people are experts (and voting) in less than 1% of the tags that a site is about...)


The absolute rate of voting is very low

This is related to the low voting activity in some sense: The influx of questions on stack overflow is too high. Cross Validated is also one of the larger sites, but still, a question may be on the front page after an hour. On Stack Overflow, the time for a question to remain on the front page may be less than 5 minutes.

(The fact that the voting activity basically drops to zero after a week is nevertheless concerning)


Inconsistent voting as function of time

I don't think that a more "nuanced" voting mechanism (beyond +1/-1) could help here, unless there was an accompanying update of how the votes affect the visibility of a question.

Right now, the popularity plays a role in two regards: When I scroll over the main page, and see a question with a title like "What causes this error in JavaScript?", I'll definitely not read it... unless it has 10 upvotes, and my geek-curiosity is triggered. The other aspect is the "10k mod tools". When a question has enough votes to appear in the "Questions with extreme votes" list, it's bound to attract further votes (usually, in the direction that caused it to appear in the list in the first place).

(Beyond that: I had a look at the queries here, and am not sure whether the times that you mentioned are right. Specifically, I wonder about the scale of the x-axis and how you came up with "half a year", considering that the query with DATEDIFF(hh, ...) seems to be about hours, but I'm not an expert here. Also, some of the queries don't directly generate the graphs as they are shown in the question, so I hesitate to say something about this)


How can we improve it?

The ways in which the voting system on Stack Overflow in particular is distorted are, (subjectively, but roughly aligned to your observations) :

  • There are too many different topics on Stack Overflow
  • There are too many questions asked and answered too quickly on Stack Overflow (and some aspects that you mentioned seem to be related to the "Fastest Gun In The West" problem)
  • Recently, people talked about observing "Pity Upvotes", where things have been upvoted to counter downvotes...
  • A severe problem: My impression is that many people are upvoting crappy questions because they can (quickly) answer them. Bad questions and bad answers are then "validating" each other - basically as a "temporary, two-person voting ring" - but nobody except for the asker and the answerer would vote for that later.

We, individually, cannot improve that - only by voting consistently. Further measures (for example, higher standards (as in "more elitism")) are beyond our control.

  • Your 'low voting activity' section seems to imply that I made some comparison between Cross Valdiated and Stack Overflow, and that there is a meaningful difference. I actually find the comparison between Stack Overflow and Cross Validated such that the difference is small. They show very much the same trend. I often do these sort of stats on both just as a verification that the trend is more general and works also in the bigger SO population and is not just an artifact for Cross Validated. – Sextus Empiricus Nov 19 '19 at 21:39
  • One would expect that 'the absolute rate of voting' does not drop when people still keep looking at old questions. Sure, it might be that a single question may receive relatively less votes... but all questions together (the absolute rate) are getting less votes. This is not so much because the site is flooding with new questions making old questions less easy to find. The number of votes on new questions are not replacing the number of votes on old questions. It is the type of activity that has completely changed. – Sextus Empiricus Nov 19 '19 at 21:45
  • 1
    @SextusEmpiricus The number of tags, users (and therefore views by logged-in users) is certainly vastly different, and I think that the topics of CV are even more concentrated within the fewer tags. I cannot quantify the influence of all that to the observations, but maybe a comparison involving other "non-flagship-sites" (dba, unix), or a "semi-flagship-site" (serverfault) could show whether this is a problem of SO, or conversely, an (unexplained) benefit of CV. (But of course, this would go beyond the core points of the question, which are valid regardless of that) – Marco13 Nov 19 '19 at 21:47
  • 1
    The trends are very universal, except... the query for ask ubuntu is very interesting to compare. You can see clear peaks every two years in april when a new version is released. These votes are a lot on old questions. – Sextus Empiricus Nov 19 '19 at 21:51
  • What do you mean by 'benefit of CV'? I state that CV and SO are much the same in these statistics. – Sextus Empiricus Nov 19 '19 at 21:52
  • @SextusEmpiricus Sorry, then I misunderstood the point of the first graph. The CV one is a bit noisy, but I thought that one of the points was that there are far more SO questions with a score of 0. (Now, when writing this, I'm wondering how strong the dependency between "score" and "votes" is - a score of 0 may be achieved with 10 votes, for that matter). But I'll consider revising the answer at that point (unfortunately, I cannot do this immediately) – Marco13 Nov 19 '19 at 22:12
  • 1
    The questions with 0 or 1 score are most likely questions with 0 or 1 votes. It doesn't happen so often that a question has fifty-fifty up and downvotes (except on meta) and that such questions end up with 0 or 1 score. – Sextus Empiricus Nov 19 '19 at 22:56
  • Indeed the CV curve is noisier. That is mostly because there is a lower volume of questions which is more susceptible to random variance. But... both sites have on the one hand an increase in questions with score 0 or 1 and on the other hand the number of questions with higher scores are stagnating (or even decreasing; in that aspect SO is a bit further than CV, and it shows a the future of other - younger - SE sites among which CV ). – Sextus Empiricus Nov 19 '19 at 23:00
  • Interesting in the SO graph (and what is not present in the SE graph) is the little bump at the year 2015. Bumps occur roughly every year in the 0/1 score questions. The bumps are regular around the beginning of the year and maybe relate with some external yearly patterns. However in the number of questions with 2, 3 and 4 or more score these bumps are not much pronounced, except in 2015 where it is very strong. So possibly whatever happened there (maybe SE being available to China) could be recreated to give a new impulse. – Sextus Empiricus Nov 19 '19 at 23:10
5

I know I'm out of my depth here, I tried to decipher the graphs but I found them confusing.

All I know is, when a site such as Stack Overflow, attracts millions of visitors daily and the number of users between 1 and 50 rep swamp the site with boring repetitive Googleable questions, and users who are only interested in badges and points post "cheap" easy answers, the quality goes down the toilet.

The users who have the rep to vote, will continue to do so as long as they enjoy the “game”. Users who care about quality, will vote less and less as their enjoyment of the site sinks lower and lower.

The solution? Drastically, radically, mercilessly, reduce the number of questions from new users flooding SO.

Give new SO users a quota of five questions to answer. If none of these answers attract at least a positive score (+1), those users are suspended from answering further questions. If they need to post questions because their livelihood depends on it, then charge them a fee: 10 dollars gives them the opportunity to post twenty questions. Limit the answers to hi-rep users (25k) to answer these questions and pay them in swag or something or another. Watch those answers and upvotes skyrocket.

I know this answer will be downvoted to hell.

  • 1
    I like it. I feel that it Stack Overflow should generally be a platform where new users are encouraged to answer rather than ask questions. Until they have received a given number of upvotes on their answers, then asking questions should, ideally, not even be an available option. This way you only get new questions from those who have shown some capability to answer questions from others. – Rounin Nov 20 '19 at 9:09
  • 1
    @RouninsaysJesuisMonica It's actually the other way round but I'll change it because it's a better idea. – Mari-Lou A Nov 20 '19 at 9:11
  • 2
    I believe as well that quality and sustainability of the voting system is related to the increase of low rep users 'disturbing' the quality control. (Some of the effects that I describe with the graphs are however less dependent on that and relate simply to the growth in number of questions and answers, no matter what their quality is - although quality and changes in it plays a role as well of course). – Sextus Empiricus Nov 20 '19 at 9:27
  • 1
    @SextusEmpiricus as someone already pointed out, I forget who, if a question stays on the home/front page an hour or so, it stands a chance of beeing seen, and maybe of being voted. If a question by a 1-rep user appears and is on page 2 after only 5 minutes of posting the chances of it being seen, voted and then answered decreases exponentially with every page drop. – Mari-Lou A Nov 20 '19 at 9:32
  • 1
    However, I believe your suggested solution does not go far enough in restraining low rep contributor's answers (contributors which stimulate the idea that low quality questions are ok). There are two problems with it. 1) there are many contributors with few answers and it is more like the number of low rep contributors that needs to be restricted instead of number of answers per low rep contributor. 2) high rep contributors (with a small, but reasonable, amount rep) will just as well spoil the game and answer low questions. – Sextus Empiricus Nov 20 '19 at 9:33
  • 1
    Also, when this is the problem, 'too many people jumping to answer simple questions', then we should increase closing efforts (but that has problems too, anyway SE/SO has a serious problem). – Sextus Empiricus Nov 20 '19 at 9:34
4

I agree. Gaining reputation by answering questions is very often hard work. Nonetheless, I consider it essential for a large group of people:

  • New users are looking for reputation to get to higher privileges (I started my whole Stack Overflow career, because I was behind my companies proxy, and thus, back then, getting to 125 or 250 or whatever significantly improved the "SO experience").
  • Gamification users are looking for reputation, because that is their core motivation.
  • Even top users with other priorities (like: "I am here to answer questions by readers of my books") don't shy away from collecting reputation.

So, please: as a gamification person I didn't like the question-weight update, and any further ideas to significantly change the reputation/voting system doesn't sound attractive at all.

And seriously: you don't want to annoy those people that contribute the majority of the high-quality content.

  • 1
    Indeed the voting is good for gamification, but not for "good content rises to the top .. incorrect content falls to the bottom" (or rise and falling are only occuring with very little nuance). So indeed, changing the system may be a too large step because it changes the entire game which is not nice for those that are playing it. However, more subtle changes can be made (Like Iamnotthewayyouspeak says in h* comment under h* post, it could be possible to not display scores directly and not apply ordering. This is also how I watch the answers, I mostly order them by activity and not by score) – Sextus Empiricus Nov 19 '19 at 20:00
  • @SextusEmpiricus Very much depends. On SO, I typically start by "highest" first. And only if that doesn't resolve the problem I am researching (and there are plenty of answers), then I will change to "active first". Especially for questions that are around for more than a few weeks, the highest voted answer does do the job. – GhostCat salutes Monica C. Nov 19 '19 at 20:15
  • on stats there are several old questions with more recently added questions where it is very interesting to see (and vote for) those new additions. Sometimes an older answer is not that great. (sure it may solve the problem, but the newer answer might explain it much more elegantly and provide more insight; stats is also about getting the problem and understanding it well and not just about finding a solution) – Sextus Empiricus Nov 19 '19 at 20:20
  • Here's an example involving myself stats.stackexchange.com/questions/429526/… – Sextus Empiricus Nov 19 '19 at 20:25
4

The voting system can be sustainable provided SE is able to attract and retain knowledgeable subject matter experts. A loose working definition of such a person might be someone who is capable of hitting 20k on a medium volume site.

A voting system built around usefulness and quality of questions and answers absolutely requires such people participating, otherwise the general trend is downwards, in both quality and quantity, and SE will eventually become what its founders set it up to oppose.

The voting system doesn't need to change; SE does.

  • 1
    I do not see so directly how these knowledgeable subject matter experts are gonna make a difference in the voting. The experts only fill a tiny fraction of the users (yes, they make most answers, with the majority of the answers coming from a tiny fraction od users data.stackexchange.com/stats/query/1156081/… but for voting this may have much less effect, those experts are not gonna see every single question ) – Sextus Empiricus Nov 19 '19 at 20:11
3

It seems our current voting system is like democracy; it's not the best system we can create, but all others are worse.

Now, it does not mean that we cannot improve the current system. I, for one, do not like the valuation increase of points for a question. Questions are easy, good questions are harder, and good answers are hardest. The previous system handled that better than the current.

What we might need is a way to devaluate older answers and questions. Good and old answers are like a nice bank account with a good return rate: you can still get points from it without doing any work. But not sure on how to make that work.

In the end, you want to reward the current answer givers and honor the previous ones.

  • "It seems our current voting system is like democracy; it's not the best system we can create, but all others are worse." To me this sounds mostly very catchy. I am not at all sure whether this is true (at least it is not well founded and slightly fallacious). Democracy is not always the best system (it depends on where and when it is applied). The voting system has equally problems depending on the era and place. What we see now is that the system grows out of proportions and for most posts the voting becomes a faint sign. Most posts only get 0,1,or 2 votes. Most posts arent differentiated. – Sextus Empiricus Nov 21 '19 at 9:36
  • I have added an additional graph to the question. It shows that the voting system is doubtful as being similar to democracy in terms of 'all other systems are worse'. What the image shows is that the voting system is probably just as worse or even more worse (except that the voting system creates gamification, but in terms of the point 'good content rises to the top...incorrect content falls to the bottom' it does not do much good). – Sextus Empiricus Nov 21 '19 at 10:15
  • 1
    It's like a defunct democracy (It could be the least worse when it worked the way it was supposed to work). – Sextus Empiricus Nov 21 '19 at 10:17
  • @SextusEmpiricus, I am not sure my equating SE voting and democracy really is true. Systems are build to get a certain result. The wanted result from SE is different then from me; I want questions answered, SE wants more money, users and be more inclusive. I hope they don't exclude one an other. – Flummox - don't be evil SE Nov 21 '19 at 10:22
  • (I consider it in this case as an analogy being equal with 'the least worse' and not with 'democracy' because voting here is indeed not democratic) Very often people come up with the "the-least-worse-argument" . However, this is a bit pooh-poohing away any criticism that is more like going into the direction that the democracy(/voting system) is defunct or not present and are about improving the democraticness(voting) of the system (you could say it is a bit fallacious like: A argues corruption is bad, then B says yeah but it's democracy, it's the least worse) – Sextus Empiricus Nov 21 '19 at 10:28
2

The idea is that voting means the best answers get promoted, and the reputation system attracts experts. But experts do not affect the voting.

Check this comment to an answer - 18 upvotes to it, yet 786 upvotes for the truly bad answer, and it is still accepted as correct. (It's far from the only one, but it is possibly the one that is provably wrong due to Eric's involvement).

Look at Eric's answers, 124 pages of answers, only 4 are in the hundreds. Voting is therefore all about popularity and herd instinct, not quality. People will uptick answers that look good, or already have lots of votes. They will also downvote similarly.

The answer, perhaps, is to make voting harder. To vote on something you must also do something to make your vote worthwhile, comment or something to admit you think it's a good answer, and thus stop the mindless upticking because it already has lots of upticks. I doubt such a practice would be workable or preferred however.

  • 3
    It is like democracy. Mostly it sucks, but we don't have better system. Problem is not in voting and reputation system on their own, it is more that we don't have any additional tools to ensure really bad answers are either removed or marked as bad (beyond the comment) and similar. Also this is why it is of utmost importance that old, knowledgeable users stay and contribute if nothing more then to help cleaning up such things or commenting at least. But everything SE has done lately is going in completely opposite direction. – Resistance Is Futile Nov 20 '19 at 7:14
  • 1
    @gbjbaanb Here is a sql query that tracks the development of the different answers. In this case the increase is roughly growing as the square of time 'score ~ constant*time^2' and the score of the different answers will start to grow further apart (and new answers will have little chance). I am sure that there will be other interesting questions with strange growth in growth for the different answers. – Sextus Empiricus Nov 20 '19 at 14:14
1

Allow an explicit blank vote as well

This change to the voting mechanism which will:

  • Not change the game-play of the gamification that voting brings (reputation will still be based on the +1 and -1 votes).

  • Yet, it will be able to provide extra information and make the votes more meaningful in differentiating quality of questions.

By allowing to vote blank (explicitly) a contributor can tell that they have read an answer/question and that they did not care about up- or down-voting. Normally we read a question/answer and either vote or not. But when we do not vote, then the voting-system does not know that (The system doesn't know whether contributors have made the consideration of voting or whether they just not yet read the questions/answers).

With the information of blank votes an algorithm can verify whether or not the questions and answers have been equally read (and possibly create some additional - more nuanced - score value aside from the current one). It can do this especially well when contributors are motivated to cast votes on all answers to a question (because then a repeated measures comparison can be made) and when they are notified of questions where their votes are incomplete (such that users can re-evaluate when new questions are added).

The advantage of this system is that it is only a small adaptation and does not really influence the up/down +/- 1 scoring and related reputation system. Already now people are implicitly voting blank. The difference is that now this can be done explicitly and the system might try to stimulate voting.

The disadvantage of this system is that, as simple as it seems, it may not be so easy technologically and requires some changes to both front and backend of the website. Also, whatever the special algorithm is going to do, it might be somewhat opaque how the score is created (although it does not need to be difficult).


Change the expressions of the score on questions and answers, by some transformed number.

This may have the psychological effect that large differences are being reduced. Users may need to be guided in understanding that a difference between score 0,1,2,3 is a much larger difference than the difference between score 10 and 100.

The effects is that the scores will be easier to compare, and in addition the contributors/voters might be having less tendencies to automatically vote on the top answers ("if 100 people or more before me found this a good question then I should find that as well")


Make voting more meaningful to a person, by linking to recommendations or other site functions.

SE/SO could place algorithms based on user votes that help to suggest new interesting questions and answers for users.

Then users might be more motivated to vote in order to get better recommendations. The advantage is that users will be considering their voting more carefully and it will more accurately resemble what users find good or not. The disadvantage is that users will be voting too much on their interests and not necessarily so much on the quality.

-2

Allow No-Account visitors to somehow Vote or contribute Reputation

and / or

Gain Reputation based on Views



The relative rate of voting is very low

The absolute rate of voting is very low

[...] the idea of a database of questions and answers (where questions are being reused and helpful to others) is getting less strong. With this fast pace only recent questions activity, the platform runs the risk to turn into a helpdesk (for quick and dirty answers) rather than a knowledge base (for high quality information).


I have several 75,000-view Questions on Stack Overflow. The Questions each have only ~100 points. The accepted answers have ~100 points. The remaining good answers attract a handful of upvotes, 5-30 points.

It seems clear that MANY people are being helped by those questions and answers. But it appears that only ~100 have been.

Likely lots of Google SERP-based visitors. (can we break down View stats of by account vs. no-account?)

Find a way to reward highly-viewed search result Questions / Answers that does not involve upvotes from people who have an account and the sufficient privileges to vote.

Perhaps allow anonymous users to upvote, perhaps at a lower-per-vote-rep. Or begin rewarding reputation for Views.

Encourage no-account voters to upgrade to an account, using "don't lose all your anonymous votes/content!" as a better upsell than the existing workflow.

Something would have to be done to normalize HNQ traffic and other external high-traffic links.

  • 3
    What I take from this is that 'increasing vote rate' is a good idea. However, I believe that the increase by allowing anonymous users to vote is very much prone to cheating with 'fake' votes (since the game element is so strong people are gonna do that), – Sextus Empiricus Nov 19 '19 at 20:17
  • 10-rep per day reputation cap? No actual user "reputation" gained, but DO increase the counter on the Question/Answer? There would be many ways to combat abuse if the root benefit was desired. – pkamb Nov 19 '19 at 20:24
  • 3
    Perhaps an anonymous vote is worth a fraction of a regular vote. Have to be careful when picking numbers because it's not just about the reputation but also the displayed score (which is advertised to other visitors as a sign of quality). You don't want to amplify e.g. the reddit effect where a publicized post attracts a lot of external attention in a short amount of time, throwing the votes out of whack with site norms. – Troyen Nov 19 '19 at 20:24
  • @Troyen So...the HNQ? – Bryan Krause Nov 19 '19 at 20:49
  • @BryanKrause Right, all the same negatives that come from HNQ exposure, but magnified because it's a much larger off-network audience. – Troyen Nov 19 '19 at 21:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .