Introduce a better way to identify trusted users

Question

Stack Overflow depends heavily on moderation by a body of reliable users dedicated to the site rather than merely gaining points and badges any way possible. It's vital to be able to tell these groups apart from each other.

Over time, reputation is going to become less and less reliable a way to do that. The voting system is easy to game, in many ways and shades. There are contributors rising "through the ranks" whose main motivation is not creating a high quality resource. They just post mediocre stuff in popular tags. Eventually, they will reach hundreds of thousands of rep.

The recent problems with the review queue show that there should be a better way of telling apart users genuinely concerned with quality from those just hanging out to earn rep and badges.

Now, machines and algorithms can always be gamed. The best detectors of dedicated users who are "for real" would arguably be - you and me! The dedicated, veteran users. In our everyday interactions on SO and maybe Meta, we come across fellow users and, over time, reach some judgement of them and their actions on the site. I could probably name fifty users right away who I would absolutely trust using the review queue. I could also find fifty users who I would absolutely not trust using the review queue.

Currently, there is no way for me to explicitly tell the system that I trust or distrust these users. Voting doesn't work for this: I may not be active in the same tags as them, or they may not be answering much on SO in the first place, but be busy editing and commenting. Or the bulk of their activity will be on Meta. Or I will know them personally from conferences and such, rather than the site itself. And one vote is only a drop in the bike-shed-ridden bucket anyway.

Should SO create a new, mostly invisible, social layer of trust? A way for a user to tell the system whether they trust a user or not. Obviously implemented in a way that doesn't break the basic SO idea of "it's about the content, not the users".

There are many ways to do this. Most are horribly flawed. There must absolutely not be any kind of public showing off of "trust rates" or something.

One way that might suck slightly less than all the others is a simple yes/no system.

A user would become a trusted user when they receive enough votes of confidence, let's call them "trust votes", from other trusted users. A number of trust votes is required to make you a trusted user, depending on how trusted the voters are. A moderator trust vote could make you trusted immediately. The algorithms used to calculate trustworthiness, and the threshold at which you yourself become trusted, would be secret. Votes of confidence would remain anonymous, so you never know how many people voted for you and who.

Once a user becomes a trusted user, the words "trusted user" would show up in their profile; no other public recognition is given.

The trusted user status gives them access to certain moderation tools that are closed to everyone else, even 20k+ users. (To me, the review queue should be one of them.) New moderation tools could become "trusted only" instead of 5k+, 10k+, 20k+ tools. (Migrating some existing privileges to "trusted only" would probably be too unfriendly an act to those who'd stand to lose it that way.)

Users would have a "I trust this person" / "I do not trust this person" button when they visit another user's profile. A trust vote is an important thing and people would have to be encouraged to use their votes wisely. Maybe through a total limit that you have to manage carefully, and that grows very slowly (say, five new votes a month).

As to how to get this whole thing started: at first, give, say, 50 trust votes to a limited circle. The ideal circle would be the people who have undergone the most scrutiny by the community, and are best equipped to have a good impression of users' activities: the moderators.

Let it spread from there, like a ponzi scheme, moderators trust-voting users, and those again trust-voting others - although maybe with less votes than the initial allotment that the moderators get. This would my favourite method of starting it because the feature would remain very low-key this way.

To reiterate, this is as non-social as can be. There is no public display of how many people trust you. You never know how many people trust or distrust you, and who they are. All that is visible to you and the public is the info whether you are a trusted user.

Making sure that only other trusted users can make you a trusted user help prevent "awesome, this user gave me the codez so I vote to trust him in return" situations. This suggestion is explicitly designed to battle bike shed voting by limiting the number of people who can vote.

Things that would have to be addressed:

• Users from tags with little traffic might go unrecognized, as there are fewer people to observe and appreciate their activity.

• There would have to be the possibility to withdraw "trusted user" status. Most likely if you gain a lot of negative trust votes. This would have to be dealt with carefully.

Answer 1

Once upon a time, votes for questions and votes for answers received the same ten points. Time passed, and an ever-growing number of people were reaching important rep thresholds based on questions -- and, to make it worse, often they were crowd- (or Fiddler-) pleasing fun or fluff.

Two responses ensued: less reputation for question votes, and clearer guidelines encouraging closure and deletion of fluff.

However, rep, like entropy, is ever-increasing. Even with less points for a question, a user who just posts a lot of fair-to-middling questions will eventually reach any threshold.

This question, however, challenges a more basic assumption. A core theory of this site was that editorial privileges should be gated by expertise, as measured by rep. Give or take the edit review system, which only functions at the low end, janatorial efforts are rewarded entirely by badges, while janatorial privileges are awarded based on content.

Changing this would be radical, in the original, root-vegetable, sense of that word. I'm afraid that this proposal would add a confusing mechanism that might not, in the end, fix things.

I submit that 'trust-votes' are, essentially, a continuous, larger-scale, moderator election. As such, this proposal should be considered, pondered, and perhaps modified in the light of expanding the moderator system.

The problem I see is that the actions that should feed an evaluation of granting privileges are not very visible. What users can see of my actions are my answers, comments, and edits, and close votes. They can't see my flags or plain old votes.

Ideally, the system would observe my proposals for actions (e.g. flags), and grant me autonomy based on whether my proposals meet with the approval of the mods.

In other words, instead of asking users to trust me based on questionably relevant evidence, the system should decide to trust me by measuring entirely relevant metrics. If my flags are nearly always right, I should be trusted more to do these things. I don't claim to have a completely worked-out proposal here, but I submit the general idea for consideration.

Answer 2

I began noticing some very odd voting shortly after the new review system was implemented. Normally, when I see a completely craptastic post with several up votes, I run straight to our tools to see if I notice any suspicious patterns. After chasing half a dozen geese, I realized that it's just people blindly voting in review.

There's also more users that write .. less than awesome posts now reviewing .. less than awesome posts. So it's not always blind gamification, ignorance is partially to blame.

What remains is, those that can maintain high quality and do participate in review flag this stuff, and we take action. So a partial solution, or perhaps the means of getting data to come up with a solution is this:

Track the amount of negative actions a moderator or trusted (20k) user took on a post any user positively reviewed, by the user(s) that reviewed it.

This should quickly be able to tell us who is doing it wrong, which leads to a better idea of what (if anything) could be done about it. I really want better data on the problem before anything is done to correct it.

Answer 3

The idea is interesting, but the thing is, people actually vote on posts. People spend a great deal of time going to posts and they generally remember to upvote them if they're good; it's a pretty natural interaction on the web. Going to people's profiles after getting certain magic votes and giving special magic person votes? Let's sanity check that quick.

People don't visit profile pages often. (Limiting factor 1) Profile pages are really busy, so a new button would be hard to notice (limiting factor 2) and some major education stuff would be needed (limiting factor 3). To boot, only "trusted users" are allowed to vote (limiting factor 4). If there's only so many trusted users, it's really easy for the system to break down if just a few people don't vote. And what if we're talking about Stack Overflow? Suddenly tag partitions are a big problem; trusted users like John Skeet and Eric Lippert might be great at identifying Trusted users in C# (if they care), but they're unlikely to see trusted users in tags they don't use, or worse, ignore (limiting factor 5).

The suggested method of starting this off (moderators voting) sounds bad too, in no small part because of the potential for drama. Suddenly I, as a moderator, am supposed to decide who gets privileges on the site. I'm sure you see the potential problem there.

On top of all of the things that drastically limit the likelyhood anyone will actually vote here, suddenly we're voting on people, not posts. With the exception of moderator elections, that's a pretty big paradigm shift.

The problem really shouldn't be judging the quality of users directly and subjectively like this. If anything a user's actions should be judged. If a user consistently approves edits that are rejected or makes edits that are rolled back, maybe they should get a notification or maybe even a brief approval/edit ban. But the whole thing where an extremely limited set of people are expected to vote on an extremely limited set of people via an extremely rare action just seems like an extremely good way to make sure we just plain don't have anyone that can actually use these tools.

If there's a problem with how people use a tool, address that. That's why we have edit bans, require multiple approvals, multiple close votes, etc. Just making a system so that virtually no one can ever use it just makes tools useless and forces the burden back on moderators who are already spread thin. The whole point of community trust and community moderation abilities is to lighten the load of moderators and make moderation a thing everyone can take part of, not just the 0.0001% of people who actually recieved or cast these new magic votes.

Answer 4

I like the idea. As jokerdino says there ought to be some mitigating factors but I would like to address the potential problems. I think they can be, at least partially, solved by the bit you edited out about allowing everyone to vote and what remains, i.e. starting with a trusted circle.

To avoid the "circle jerk" phenomenon I would start, as you suggest, with the moderators. Give each moderator unlimited votes. Once a moderator has nominated you, you're automatically "trusted".

Every trusted user then gets a limited number of votes. For example, 12 to start and then one a month. The point of this is for it to be a low number. People should be jealous of their votes so they're less likely to use them unwisely. It would make sense for the first batch to have more votes to spread the love around a little more.

It takes X votes from trusted users in order for yourself to become trusted, say 2.

Then to up the chances of ensuring that all tags are covered you allow everyone to vote. To stop everyone becoming trusted you still have to be nominated by one trusted user. You also need Y votes from other users. In other words you become trusted if you have the following:

• Y votes from "ordinary" users and one vote from a trusted user.
• OR X votes from trusted users.
• OR 1 vote from a moderator.

There could be a little list somewhere of people who need a single vote from a trusted user in order to become "trusted".

A trusted user would also get 1 "untrust" user vote each month. It would require another Z votes from trusted users for someone to become no longer trusted. Moderators would be included in this but would have unlimited votes (no accusations of revenge attacks).

Where I would differ from you is that in time, I think it would be worth applying this to the review queues (in the future). I'm not saying that you have to be a trusted user to use them necessarily but that you would have to have been voted as a trusted user by a few not yet trusted users in order to use them.

Answer 5

Personally, I think this idea on a whole is misguided.

The purpose of reputation is to fairly and dispassionately measure "trust" for how SE defines it. With this system, we might as well scrap reputation entirely. And it sets up an awful precedent: if certain people don't like who is at the top of the proverbial "trust" meter, scrap it until we come up with a system that matches those people's truthy idea of who should be trusted.

But I wanted to highlight one part of your proposal:

As to how to get this whole thing started: at first, give, say, 50 trust votes to a limited circle. The ideal circle would be the people who have undergone the most scrutiny by the community, and are best equipped to have a good impression of users' activities: the moderators.

One of the great benefits of Stack Exchange and how it even bills itself is that the community runs the sites, not them. Why would you cede this amount of power to a set of people who are elected for life?

For that matter, any time the moderators are asked to do a little more, they push back saying that they don't have time or that they're overworked or that they're just volunteers. Why would any moderator agree to this plan? Having to be solely responsible for deciding who is trusted on a site? If a moderator is overworked or has better things to do, this seems like a real hassle.

I get that you don't like how the review queue is going; I really do. But effectively scrapping the reputation system and ceding a huge amount of power to a select group of people is not the solution.

Rather, the problems you describe can easily be attributed to two things:

• Tools that could use some improvement, because it's too easy to do bad stuff and too difficult to undo aforementioned bad stuff
• A largely disinterested user base who haven't, as yet, found a reason to take the review queues as seriously as people like you do

Instead of writing off the tools and creating a far smaller pool of users helping with community moderation, I'd much rather see the tools improve and ideas on how to get the community more engaged.

Answer 6

Trusted users are probably, and hopefully folk who have put in a certain degree of community involvement. Considering that one of the things that @Pekka is proposing for the "trusted user" status is having access to the review queue, could I propose that one of the criteria for this would be a reasonable number of helpful flags (say 50), and at least a 70-80% ratio of helpful to non helpful flags - this means the user takes the effort to flag (and 50 isn't that hard).

And while using mods to 'seed' trusted users is a good idea, I think allowing a mod veto/revocation of the trusted user status would be a good balance against it. A large number of negative flags (of unknown amount) by distinct users might be grounds for a review of the status (automating it somewhat).

What I find problematic is the blackbox nature of this. While it prevents the system from being gamed, it feels someone against the whole open nature of the site. Its something that needs to be balanced pretty well.

Answer 7

I would like to propose one alternate solution how about this instead of trusted user ?

Keep the review system, allow a user to review a post if he has earned a silver/gold badge related to a question. For ex: to review a C tagged question I need to have a C Silver badge. That way not everybody can review it, it puts additional restriction.

PS: Do let me know reason if you choose to downvote.

Answer 8

I really like the idea behind the suggestion but I would like to propose some minor alterations, which I guess would make the idea a teeny bit better.

• The trustees should not have a permanent seat like the moderators. This would help us control the damage in case any user goes rogue and undetected. Each user may have a term of 30 days or so with any significant lapses of judgment costing their position.

• Going with your original idea of allocating votes to already trusted users, any user who accumulates a certain number of anti-trust votes cannot be appointed as a trusted user for, say 6 months or so. If someone is not trusted by quite a number of people, that should be a good enough indicator.

• Given subtle hints that the community team is overwhelmed with monitoring all 300+ community moderators across the network, it might be wise not to grant insta-votes to trusted users. Instead two votes from any trusted user can be considered final, which would make the body of trusted users self-regulatory and a safer solution.