Can we use something other than reputation as the bottleneck for election primaries? [duplicate]

Question

We recently began the sixth moderator election on Stack Overflow. There is a huge number of highly qualified candidates, and there are 23 overall at the time of this writing.

That's great, but there's this to consider:

After 7 days, the top 30 nominees, ordered by reputation, advance to the primary phase. However, if there are 10 candidates or less [sic], we skip directly to the election phase. _Source

Considering the fact that it has been less than nine hours and there's still a week to go, it is almost certain that we will hit that 30 candidate threshold by the time nominations close.

Since we mostly agree that reputation isn't a very good indicator of moderation ability, I really don't think that it's a good idea to use that as the bottleneck for primaries. This is really only a problem on large sites like Stack Overflow, but it is a problem nonetheless.

The major issue here is the fact that there are a lot of high rep users who will come seemingly out of nowhere for the moderator election and disappear again afterward. A lot of these users, despite often having high (>= 20k on Stack Overflow) reputation, have very low Meta participation and very little moderation experience.

Because of this, they'll be pushing out users who may be more qualified in terms of moderation experience simply because they have more reputation, which many herald as a sign of site participation, but which can also be the result of many other factors.

For that reason, I don't think it's a good idea to use reputation as the deciding factor here. This is a time-sensitive issue, as the primaries start in a week. It appears that this will be the first election since, well, the first election that the bottleneck will be invoked, and I think we can all agree that a lot has changed since then.

Answer 1

I almost proposed merging the nomination and primary parts of the election after almost not making it through to the primary in 2011 myself.

Had a curious passer-by not dug through my history and noticed that I dropped a 500 rep bounty (we used to toss in +50) on a question that really belonged on meta anyway, I would have been knocked out of the primary. I could tell that I was getting more support than users with way more rep than I had, but there wasn't any way for them to express their support in a meaningful enough way to ensure that I advanced.

The problem with making this decision based on any score is that you potentially alienate very good candidates. They either don't get a chance to move on after standing their ground in the nomination phase, or they (worse) don't bother entering the nomination phase because math tells them it would be pointless to do so.

The only way to really fix this is to allow the community to decide on who moves forward, and the only way to do that is to let them vote. The outcome, however, still has to be the same - you can't get into the actual election where you get to cast your first / second / third choice on a list any larger than 10 - decision paralysis and all. We're seeing increasingly better turnouts in elections, any move that reverses that would be a bad one.

It's not as easy as just merging the two phases, there has to be a period where candidates just communicate with folks that have something to say about them nominating, or their nomination. It's important to give candidates this opportunity, even if it leads to them realizing that they didn't quite realize the level of scrutiny they were taking on.

This is something I really care about, for pretty obvious reasons. I want to defer it until we can study how the new candidate score might have influenced all sorts of things in this election. I think we need an outcome and some control data from elections on a few smaller sites until we decide on what to do next.

It's not an easy problem to solve. I've had more than several ideas, but I want to let this (and possibly a few more elections - fear not, they're coming / running shortly on other sites) play out first before I propose anything.

Answer 2

One possible solution to this is to use the new candidate scores to weed out candidates.

However, in its current form, the candidate score still has a few issues:

1. It encourages gaming during the election period. Making the bottleneck based on candidate scores would only encourage this more.
2. It's still heavily skewed toward reputation. 20 out of 40 possible points are earned from reputation (which is floored rather than rounded), so in essence you have the same problem. However, this does decrease the effect of reputation to a certain extent. The main problem here is that the candidate score metric is not detailed enough.
3. It doesn't take into account a lot of important moderation factors. Among these are Meta participation (you could make an argument for the Quorum and Convention badges fulfilling this, but they're not weighted heavily enough and are fairly easy to earn.), flag counts (an old metric that I think still has merit despite its tendency for gaming), and time spent reviewing in the review queues.

So, while I think candidate scores are a decent metric to be used here, I'm holding out for a better solution.

Overall, I really think that the problem here is with the bottleneck process itself. Simply stripping candidates before the primaries is a bit reckless; there has to be a better way to make decisions about who should be given the chance to run.

Answer 3

Reading the question reminded me of something Joel wrote about interviewing:

On the other hand, if you reject a good candidate, I mean, I guess in some existential sense an injustice has been done, but, hey, if they're so smart, don't worry, they'll get lots of good job offers.

There will be more elections in the future and there are plenty of other sites where people might be appointed or elected as a diamond moderator. If our stupid criteria eliminates some ideal candidates, that's unfortunate, but not tragic.

The nomination period is actually a lot more like sorting resumes, though. During the primary phase we will be asking tens of thousands of people on Stack Overflow to have an opinion on 30 candidates. That reminds me of the exercise I used to do everytime I had a meeting at the previous job: estimating how much the meeting costs. If each nomination take, say, 1 minute to read and all 21,571 people who voted have one more nomination to consider, that works out to 15 days of programmer productivity. Yeah, it's a meaningless calculation in oh so many ways, but my point is that we owe it to the voters to narrow down the list somehow.

That said, I like your suggestion of swapping reputation for candidate score is a good one. Reputation alone mostly captures a candidate's investment in the site. The other components get at a candidate's aptitude and enthusiasm for tasks that are more typical for a moderator.

Answer 4

I totally agree reputation isn't a good factor to filter on. Some users are here for a long time, accumulating reputation from old posts that got a lot of votes. That's totally fair, from a 'how useful was this post' point of view, but doesn't say anything on the moderator capabilities.

Some factors I had like to see:

• Give extra weight to the gold badges related to moderation, like Marshal, Steward, etc. Maybe on SO and other larger sites, make this the minimum required badges before allowed into the nomination, or give them extra weight in the candidate score.
• Give a 'meta score'. Since we don't have rep on meta, it is hard to see how the user's opinion on the community is appreciated by the community. We could show the number of Nice Question or Nice Answer badges, or a total score as if there was rep on meta.

Answer 5

There are already a number of heuristics in place to make sure people who aren't prepared can't be nominated. I'm not eligible - I don't have all the necessary badges.

Once that's taken care of, you have great people who will be moderators.

Here's where reputation comes in: In every past election the votes have always skewed heavily towards those with higher reputation even after reputation was hidden in the elections.

While we might like to bandy about the idea that reputation doesn't mean good moderation, the reality is that people vote more heavily for higher reputation users.

So this heuristic is in place not because reputation is an indicator of moderator success, but is an indicator of election success. Those with the lowest reputation in the cohort simply do not get elected. So it's best to weed them out early.

32 people is plenty of people to vote on. They could be chosen randomly from the nomination for all it really matters - but it makes sense to pre-select the strongest candidates, and voting history shows us how to accomplish that goal without inserting yet another voting step - which would simply mimic the same process.

Side commentary

Look, let's attack this whole "reputation doesn't mean a hill of beans" thing head on.

The reality is that those with high reputation have it because they enjoy being here, and they have the skills to interact favorably with the rest of the community.

Mostly, the writing skills, and the ability to understand another person, then help them resolve their problem.

In other words, their reputation is actually a better indicator of how helpful they are and how much time they spend here than of how expert they are.

Maybe:

expertise != moderation.

I believe that there's a strong argument, however, to be made for:

high reputation == helpfulness
high reputation == time

Both things that are required for moderation.

I don't think choosing by reputation to be the evil spoken of in the OP.

See that bolded part above? I suspect that the same helpful, congenial writing style that attracts so many upvotes towards reputation is evident in the election writings. Whether it's charisma, or obvious "good fit" with the established community "style" is irrelevant. Reputation, at least indirectly, measures how well this person interacts with and fits in with the rest of the community, which is more important to moderation than a lot of other, poorer heuristics.

Answer 6

One thing that might be good in the mix (both here and in other areas) is some measurement of successful moderation actions, ideas including, among other things:

• % of edit reviews where your review agreed with the final decision.
• % of flags helpful.
• % of close votes on questions that were actually closed (and perhaps not reopened).
• % of delete votes on questions that were deleted (and perhaps not undeleted).
• Etc.

I'm not sure how the various components would be weighted among each other and along with reputation (if that's still included), badges, etc. but maybe they are at least somewhat decent indicators of how various moderator actions reflect the community's norms.

Some minimum number of each stat considered here would have to be required, though, to reduce noise.

Also, perhaps only recent (say, within the last year) activity could be considered, as people tend to learn and improve over time (and also this filters out, say, somebody who was very active a few years ago then disappeared).

I don't necessarily think this is the best idea (esp. see Servy's comments below) but just throwing something out there to think about.

Answer 7

The problem is two-fold: you don't want to rule out good candidates who are below the rep threshold, and you don't want to have so many candidates that the community can't reasonably evaluate the field. But you might be able to still have both by changing the process. (I am loosely adapting what follows from large martial-arts tournaments I used to participate in.)

If you have, say, 50 candidates, it's not realistic to have everybody vote on all 50. You need an elimination mechanism. This is why there's a primary to winnow 30 down to 10, but your problem is that you need to winnow a larger field down to 30. But you can't do it like the primary, because then everybody has to review every candidate and you may as well just have a huge primary, which is suboptimal.

Instead, allow candidates who would be kept out of the top 30 (by reputation) to challenge in. A candidate may choose one candidate (who is in the top 30 and who has not already been challenged) to directly compete with for the slot. A strong candidate who isn't high-rep gets a chance to compete, but everybody doesn't have to look at all candidates.

This should be a quick, lightweight, head-to-head runoff; you don't want to delay the rest of the election too much and you don't want to open it up for a lot of extra campaigning. You should've made your case already. This is just a first cut, but perhaps something like this:

• This phase runs for 24 hours, commencing with the close of nominations.
• A candidate who is "below the line" can click a "challenge" button on a candidate who is above the line -- first come first served; the button goes away once it's been used.
• The election page displays (perhaps in a new tab) the nomination posts (with comments) for both together, with a vote button for each. (All challenges would be shown on this page.)
• Anybody who's eligible to vote can click on exactly one of the two vote buttons. Alternate idea: A single voter can vote in only one challenge, to contain the "too many candidates to review" problem (pick the race you most care about). Either way, some subset of the community votes between the two.
• Either candidate can withdraw during this phase, ceding the slot to the other.
• 24 hours after nominations closed, the winners of these challenges and the unchallenged remainder of the top 30 proceed to the primary. The primary can be shortened by one day to keep the overall election cycle the same length; think of the challenge round as "pre-primary".

Yes I know that this breaks down if you have more than 60 candidates. Gotta draw the line somewhere; if you have more than that, some of them just aren't going to get a chance.

I don't have a horse in this race. I just enjoy trying to solve systems problems like this.

Answer 8

The biggest problem with using reputation as a standard for candidates is that it doesn't actually represent the reputation of the candidate. Your reputation is what others think and feel about you; the SO reputation is actually a record of your activity on the site.

The closest thing to true reputation is the bounty system although it would be great if you could freely assign bounty to people that have been helpful in editing your posts or providing helpful comments in the manner of a tip. Of course there is the risk of gaming the system but that's just part of the site...

The current reputation system is also slightly skewed towards long-time members, you could have achieved 40/40 for the purposes of the election it would be helpful to see the average activity over the last week/month/year so that we could better gauge the recent activity of a candidate.

I feel the combination of reputation and activity would be the most helpful means of identifying the minimum bar for candidate selection.

Answer 9

While reputation most certainly does not work in all situations, I believe that it's appropriate to use reputation to sort candidates at such an early stage.

The purpose of the nomination phase is so that we don't need to be looking at so many users to vote from during the primary phase. When there are so many candidates available to choose from, the signal to noise ratio becomes even worse, and it becomes much harder to choose a candidate that will actually be suitable for the position.

Especially given that there are still 30 candidates to choose from during the primary phase, I doubt that there would not be one person from there that would not be willing and capable of handling the responsibilities of being a moderator on one of the world's largest Q&A site.

Most other metrics to measure a user's participation on the site can be gamed without other users being able to easily audit the user's activities. I'd even say that a reputation cap of 5k would be appropriate on a site like Stack Overflow, given that you only gain close voting privileges at 3k and it takes time to understand how it works.