Replacing the candidate score with something better

Question

The candidate score that is prominently displayed in elections is calculated from the reputation of a user and a number of badges that are judged to be relevant to the duties of a moderator.

It's better than just showing the reputation of each nominee, but not by much in my opinion. It's still a deeply flawed measure, and there is lots of room to improve it.

One major flaw is that the score is tied to badges, and therefore to entirely arbitrary thresholds of activity. Before the candidate score we used to show the number of flags a candidate has cast. Now we only see the score, and while it contains the number of flags in a way, it's obscuring the actual activity of a candidate in community moderation.

The badges don't scale well. They're enough to distinguish users that don't have any interest in community moderation from the users that are actively participating. But they're not suitable for anything else. The thresholds are arbitrary and fixed across the network. They might work to distinguish crappy candidates without a chance to be elected anyway from the real candidates, but they're far too coarse to distinguish between an okay candidate and a great one in terms of previous participation in community moderation.

I suggest to simply provide voters with more detailed information about all the moderation actions a candidate has performed, similar to the old statistic about the number of flags a user has cast. Flags aren't everything, and I think the following moderation activities should be shown on the election page:

• number of helpful flags cast / number of flags acted upon (alternative for pro-tem moderators)
• number of reviews (where the user was in line with the majority of reviewers)
• number of successful close or reopen votes
• number of edits to posts by other users

I think simply showing numbers is more useful than the current candidate score. This gets rid of the arbitrary thresholds inherent in badges, and properly values users going far beyond the usual expectations.

In the interest of full disclosure, the way moderation flags are handled affects me personally in an election running right now. Moderators can't earn badges for flagging, and reviewing as a moderator on a small site is often problematic. This means as a pro-tem mod I'm missing out on certain badges because I either can't earn them at all, or because I don't want to override the community by doing too many reviews. This is not a huge problem, but it is a flaw in the candidate score.

Answer 1

• number of flags cast / number of flags acted upon (alternative for pro-tem moderators)

This is what motivated the change to Candidate Score in the first place! Flags alone could easily become a source of detailed statistics about a given candidate's views and abilities. The closer you look, the better an idea you get of a given user's skill at evaluating various scenarios... And the less you see of their other work on the site.

• number of reviews

Just the raw number? Not the number of reviews that've contributed to consensus, or the number of reviews that've resulted in action, or the number of times the reviewer has failed an audit or been banned from review entirely? Not a breakdown by queue that would allow you to differentiate between someone who camps in First Posts and someone who carefully reviews all suggested edits or provides a voice of experience in closing and reopening?

• number of close or reopen votes

There's even less of a check on this than there is for flags; you can burn $SiteLimit close and reopen votes per day every day as long as there are questions you haven't voted on yet. No telling whether they're useful or not; maybe you just like clicking buttons?

• number of edits

This one at least is slightly harder to game, since it's completely public; if you're going around making tons of lousy edits, folk tend to notice. But like flagging, we could just keep going down the rabbit hole here: do you edit your own posts? Do you edit answers, or just questions? Do you add information, clean up poorly-written posts, obsessively retag or retitle? Are your edits "successful" or do they get rolled back or re-edited a lot? Do you make "heroic" edits on questions that would be lost otherwise, or do you add that last bit of spit-polish to thousands of posts that would otherwise each contribute a small broken window?

So we could have three metrics that are as problematic as what we had before Candidate Score (or worse...) and one that's maybe slightly better. And even combined, these are hardly a comprehensive view of anyone's activity on a site unless considerably more fleshed out. Nevermind that a good chunk of this data isn't even public right now; doing this right would mean adding a lot of extra information to profiles and data-dumps... This would be A Project.

So... What's the problem we're trying to solve here?

they're far too coarse to distinguish between the reasonable candidates.

That's... Ok. Once you're down to only reasonable candidates, you can stop looking at statistics and start looking at the people themselves. How well do they communicate? How well do they know the folks who will be voting? Are they answering questions put to them with skill and patience? Do they get defensive or combative when challenged? Are they picking fights with other candidates? Is the shape of their avatar going to clash with a diamond?

Once you're down to reasonable candidates, you can stop trying to read tea leaves and actually participate in the election, that messy process by which groups choose one of their own whom they can maybe tolerate giving a bit more power. That's why the Candidate Score is capped, and capped fairly low - once everyone running hits 40, you can reasonably assume they know enough about how everything works to stop worrying about giving guns to monkeys and focus on... People.