16

Has anyone considered using a PageRank-like algorithm for the reputation system? This would help to address the common criticism that users gain more reputation for witty one-liners than detailed, technical answers. (Anecdotally, I find this to be true. I got more reputation for a one-line answer to a funny question than any detailed answer I ever gave.)

So why not apply some ideas from PageRank to Stack Overflow? Google's PageRank is so successful because a link from a respected site will give your webpage a lot more rank than a link from some obscure site. In the same way that Google doesn't count all links equally, Stack Overflow shouldn't count all up-votes equally.

An up-vote from someone with a lot of reputation should be worth more than an up-vote from somebody with no reputation. Intuitively, an upvote from Jon Skeet is worth a lot more than an upvote from (unknown yahoo user), because the community trusts him.

This would also help to mitigate a lot of other ways you can game the system.

Edit:

Ah, I thought my idea was original, but evidently not. Having read the reactions to the previous person who suggested this, it is clear the SO community at large is not in favor of this idea. Still, I don't find the reasoning very solid. Disturbingly, the main line of reasoning against weighted up-voting (WUV) also implies that reputation is almost entirely useless.

The main reasons against WUV are:

1) Reputation != expertise. It is mostly an indication of how much time you spent on SO.

Really? Then perhaps we should just replace the rep system with a simple uptime-like readout. But seriously, the SO FAQ specifically says "Reputation is never given, it is earned by convincing other Stack Overflow users that you know what you're talking about". Clearly, there is at least supposed to be a strong correlation between reputation and computer programming knowledge. And if there isn't a strong correlation between reputation and computer programming knowledge, then the whole rep system is basically useless; it becomes nothing more than a measure of social standing.

But if reputation is correlated with computer programming knowledge, it seems obvious that an upvote from someone with a higher rep is likely to be more valuable. Sure, there are edge cases where this won't be the case; for that matter, there are horrible webpages with a high PageRank on Google - but I'm not looking for perfection here.

2) Reputation does not mean you're an expert, because reputation can be earned by asking masses of low value questions.

True. But the solution to this is simple. Question upvotes shouldn't increase reputation. This makes sense, because the only reason for WUV is because we should value the answers given by more knowledgeable people. But it doesn't take any expertise to ask a question. So, upvotes on questions shouldn't increase reputation.

3) Reputation is too broad a measurement; it doesn't indicate that a particular voter is knowledgeable in that particular subject.

This is probably the only significantly difficult rebuttal. It's true that a person's overall rep score doesn't indicate they are knowledgeable in any particular subject. But that doesn't mean a WUV system couldn't take specific expertise into account. In fact, the SO tagging system already records the information necessary to resolve this. For example, if you look at my user-info page, you'll see that I've answered a lot of C++ questions, but no Lisp questions. Therefore, my upvote on a C++ question should be weighted more than my upvote on a Lisp question. Adding a WUV system which takes into account expertise in specific areas would really help to better identify answers which are more likely to be valuable to the questioner, as well as everyone throughout posterity who reads the question.

4) Adding a WUV system would make it more difficult for new people to gain reputation

A WUV system will only make things difficult for new people who don't know what they're talking about. People who actually know what they're talking about, and who provide useful answers, will quickly gain reputation in a WUV system. (Perhaps even faster than in the current system.)

And finally, a WUV system in general would help to mitigate many of the ways people are able to game the system to increase their rep points, or otherwise increase their rep in ways that don't correlate to actual expertise.

4
5

The problem I see here is that you have no way of gauging the quality of a reputation score. High rep doesn't always equate to high quality.

There are other users of Stack Overflow who have lower scores but who post very high quality answers or questions in specialised areas, perhaps less often, and where the topic is of less interest than say C# Generics, Java Date Time problems or SQL JOIN questions. Who's to say that these votes are less valuable than Jon's, Joel's or Marc's for example?

Also what happens when a high rep 'trusted' user casts a down-vote? Should that down-vote also be weighted by taking into account their rep? Votes can be cast for idealogical reasons rather than technical ones and that would introduce some very odd skewing.

I can see what you're getting at but I think this is a flawed idea. There would need to be extra 'quality' data applied to this ranking or weighting system which isn't captured by Stack Overflow presently.

3
  • True, but your criticism applies equally to the current state of the rep system. And of course, in the case of Google, there's nothing to prevent ideological interests from increasing the PageRank of a site. But overall, Google is pretty darn good. Similarly, with a system based on weighted up-votes, it's more likely that over time, answers from better qualified users will be given more value, mirroring the real-world value we invest in expert advice. – Charles Salvia Nov 6 '09 at 17:42
  • We'd still need some extra data or quality screening to be able to do this. – Kev Nov 6 '09 at 17:48
  • If you assume quality = value, it is not how much effort you put into a Q/A, but how much it mattered to other people. i.e., if you spend 10 hours writing a book-quality post on how to fix a bug in a legacy COBOL mainframe, it is of less value than a couple utilitarian comments on C# from Jon. The top users may have a bunch of witty, silly one-liners where they gained a lot of reputation, but that's just because people already know them. There's noise in the system but there's hardly any malice/bots here and even with those on the web Google's pagerank still works wonders. – Camilo Martin Jul 11 '17 at 11:13
1

Why not use the algorithm described by Randall Munroe here: http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html ?

It seems to be pretty effective, and intuitively, it does what you want it to do.

1

I agree we should consider implementing a PageRank method to calculate the reputation score.

I think this will increase the reputation of expert users, who do not have much time to spend on SE, but provide very valuable input. F.ex. Herb Sutter, clearly is a C++ expert. He has only ~2k rep, but he has done so by answering only 3 questions. However, he only answered questions asked by >20k rep. users. One could expect his upvotes are also from other high rep/expert users.

On the other side, some users ask or answer many easy solvable questions, gaining a very high reputation score, f.ex blankman. His top question is Does Python have a string 'contains' substring method?. I don't think many expert users upvote this question. So, maybe such users don't deserve such a high reputation score, as it doesn't represent their actual expert status.

The PageRank will make the distinction between those valuable and crap votes. Look at how well Google Search performs. 99% of the internet is crap and bad people try hard to get their crap in the top results of Google. Nevertheless, Google's PageRank does a very good job. I'm even convinced that it could replace the entire review queue to a large extend.

2
  • The real value and interest is from the number of views (and thus view rate) on each question (mostly hits from search engines). This could be scaled/normalised in various ways: by the size (number of questions) of the most important tag (the most important tag = e.g., by highest number of questions or total number of views for all questions in that tag). The scaling does not have to be linear. The voting should also have a role, but the votes must be compensated for the compound interest effect (virtuous cycles/positive feedback mechanisms). – P.Mort. - forgot Clay Shirky_q Nov 17 '20 at 5:26
  • @p-mort-forgot-clay-shirky-q, The views/view rate highly depends on the algorithm used by search engines to show top results. It may be a crap questions and/or crap answer, even if it has a high view rate. I'm not sure it is a better measure than the up and down votes. – m7913d Nov 17 '20 at 11:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .