47

We recently began the sixth moderator election on Stack Overflow. There is a huge number of highly qualified candidates, and there are 23 overall at the time of this writing.

That's great, but there's this to consider:

After 7 days, the top 30 nominees, ordered by reputation, advance to the primary phase. However, if there are 10 candidates or less [sic], we skip directly to the election phase. Source

Considering the fact that it has been less than nine hours and there's still a week to go, it is almost certain that we will hit that 30 candidate threshold by the time nominations close.

Since we mostly agree that reputation isn't a very good indicator of moderation ability, I really don't think that it's a good idea to use that as the bottleneck for primaries. This is really only a problem on large sites like Stack Overflow, but it is a problem nonetheless.

The major issue here is the fact that there are a lot of high rep users who will come seemingly out of nowhere for the moderator election and disappear again afterward. A lot of these users, despite often having high (>= 20k on Stack Overflow) reputation, have very low Meta participation and very little moderation experience.

Because of this, they'll be pushing out users who may be more qualified in terms of moderation experience simply because they have more reputation, which many herald as a sign of site participation, but which can also be the result of many other factors.

For that reason, I don't think it's a good idea to use reputation as the deciding factor here. This is a time-sensitive issue, as the primaries start in a week. It appears that this will be the first election since, well, the first election that the bottleneck will be invoked, and I think we can all agree that a lot has changed since then.

  • 11
    Devil's advocate: While, in reality, we know that reputation != ability, the fact is all the mechanics on the site are designed to treat rep as an approximation of ability. Many community behaviors are defined using rep as an indicator, even though it is not correct. Using rep as a filter for elections, then, is consistent with the system. So while not a great indicator in reality, philosophically it seems right: If we decide on a different filter for this, then why stop here and not take it a step further by redefining site mechanics based on whatever new indicator is decided here? – Jason C Apr 7 '15 at 7:06
  • 3
    @JasonC xkcd.com/1432 – AstroCB Apr 7 '15 at 7:07
  • 3
    @JasonC You're right. Reputation is given to users based on reputation rather than moderation abilities. That's because we have no other solid metric to go by. For elections, we now do, and I think we should use it. Frankly, however, even that's not an ideal solution and I really don't think we should be bottlenecking candidates before primaries at all, but it's what we have to work with. – AstroCB Apr 7 '15 at 7:12
  • 3
    While I understand the concern, in practice I haven't seen this actually block many good candidates. This used to be a problem in Stack Overflow elections before the moderation badge requirements were added, but those badges cut down on high-rep nominations significantly. In the last couple of elections (where the badge requirements were enforced), I only think a handful of people were forced out by this. – Brad Larson Apr 7 '15 at 13:26
  • Could it not just go by the score the user has? – Tim Apr 7 '15 at 15:46
  • 2
    Surely someone (better at maths and code than me) can work out a basic algo? I mean, literally off the top of my mostly empty dimwitted head: Rep vs time as member, plus some other stuff, like total net good flags etc etc? One could be here for 6 years and have 10K rep and 1000 flags etc, or be here 1.5 years with the same stats. It can't be hard given DB access to rake together some decent method to determine what constitutes a good user. Rather than "I have 10K rep more than you - nah na na nana Thhhbssppp" etc – James Apr 7 '15 at 21:20
  • @James You say that a user who is 4 times more active will be a better mod than a less active user who has been around 4 times as long. I'm not convinced that's true. It's just that instead of "I have 10K more rep than you" it will be "I make 4K more rep per year than you". Hardly an improvement. – Dmitry Grigoryev Aug 20 at 7:18
26

I almost proposed merging the nomination and primary parts of the election after almost not making it through to the primary in 2011 myself.

Had a curious passer-by not dug through my history and noticed that I dropped a 500 rep bounty (we used to toss in +50) on a question that really belonged on meta anyway, I would have been knocked out of the primary. I could tell that I was getting more support than users with way more rep than I had, but there wasn't any way for them to express their support in a meaningful enough way to ensure that I advanced.

The problem with making this decision based on any score is that you potentially alienate very good candidates. They either don't get a chance to move on after standing their ground in the nomination phase, or they (worse) don't bother entering the nomination phase because math tells them it would be pointless to do so.

The only way to really fix this is to allow the community to decide on who moves forward, and the only way to do that is to let them vote. The outcome, however, still has to be the same - you can't get into the actual election where you get to cast your first / second / third choice on a list any larger than 10 - decision paralysis and all. We're seeing increasingly better turnouts in elections, any move that reverses that would be a bad one.

It's not as easy as just merging the two phases, there has to be a period where candidates just communicate with folks that have something to say about them nominating, or their nomination. It's important to give candidates this opportunity, even if it leads to them realizing that they didn't quite realize the level of scrutiny they were taking on.

This is something I really care about, for pretty obvious reasons. I want to defer it until we can study how the new candidate score might have influenced all sorts of things in this election. I think we need an outcome and some control data from elections on a few smaller sites until we decide on what to do next.

It's not an easy problem to solve. I've had more than several ideas, but I want to let this (and possibly a few more elections - fear not, they're coming / running shortly on other sites) play out first before I propose anything.

  • 3
    This is a great post, thanks. It's a little bit of a catch 22: If we want the community, instead of a site metric, to decide who moves forward, then the community has to make a decision on an intimidatingly large number of candidates in an early phase -- but at the same time the point of deciding who moves forward is to avoid that kind of decision paralysis. Plus all the trickiness of making sure early nominees with extra time, nominees listed first on the page, etc. don't have an advantage. – Jason C Apr 7 '15 at 18:31
  • 4
    @JasonC Exactly. It's not only an intimidatingly large number of candidates, but the time spread also factors in. The decision the community makes would need to be almost "atomic", in that the candidates see it all at once, or folks that could probably have been elected might just bail. If you look at past elections, primaries and actual elections rarely agree with each other. But you can't make a 4-day process "atomic" for anyone interested in it, unless you hide the mounting outcome from everyone. I might be onto something there ... – Tim Post Apr 7 '15 at 18:39
  • 1
    I was thinking about this earlier and also thought about hidden votes in the first phase. Is there any correlation between the order the candidates are displayed in the list and votes? How many people voted in last year's phases? Enough that if each person was hypothetically presented with a random subset of candidates in the first phase, the end result might be a good indicator? – Jason C Apr 7 '15 at 18:46
  • @JasonC I don't know because the order is just random, I don't have a way of going back and seeing it (that I know of), It is a little teeth-grinding when you know the sampling consists of so many possible things on top (or bottom). The new score helps to confront certain kinds of selection bias, but I won't really know what until we've done a few more. I'm going to shut up before inventing even more ways to say I don't know :) – Tim Post Apr 7 '15 at 18:47
  • 2
    Might I suggest culling the field by top "moderation scores" (the points out of 40 that each candidate receives), rather than just pure reputation? Seems like a very simple, zero-negative-side-effect, and easy-to-implement solution. It's probably not a cure-all, but it's definitely an improvement any way you look at it. Still takes into account your reputation, but also takes into account your moderation activity, which should be the most important factor (arguably the only factor) for being elected to a moderator position. – TylerH Apr 9 '15 at 3:31
  • @TylerH: at issue there is that there is too little resolution to distinguish between candidates. If positions #30 and #31 in a candidate-score-sorted list both had, say, 16 points, who is dropped out? Perhaps you can then fall back on total reputation, which does then still mean that people with more reputation get an edge over those with less rep but more badges (8k rep + 8 badges trumps 7k rep + 9 badges). – Martijn Pieters Apr 9 '15 at 12:51
  • @MartijnPieters Yes, I meant use the score as the primary criterion. The rep could serve as the secondary/tie-breaker criterion. – TylerH Apr 9 '15 at 13:31
  • @MartijnPieters I'm not running; that Ty person is someone else. – TylerH Apr 9 '15 at 13:48
  • @TylerH: ick, sorry, my mistake. Mea Culpa! – Martijn Pieters Apr 9 '15 at 13:50
  • @MartijnPieters No sweat. I admit (as I mentioned before), such a change definitely wouldn't be a cure-all. I do think, however, that it would be an improvement with zero additional or different side effects. – TylerH Apr 9 '15 at 13:52
  • 1
    Thanks for confirming that this is under review. There are worse things for a site than to have to choose amongst the highest rep users the few that get to empty the trash cans and sweep up the messes caused by a) honest mistakes and misunderstandings and b) delinquents - especially now that elections are tied to need / retiring moderators and not tied to a calendar. I would guess that the final answer is to weed out moderator candidates if they fall below the statistical mode of three measures. 1) rep 2) flag/badge formula 3) meta rep+participation – bmike Apr 13 '15 at 19:56
26

One possible solution to this is to use the new candidate scores to weed out candidates.

However, in its current form, the candidate score still has a few issues:

  1. It encourages gaming during the election period. Making the bottleneck based on candidate scores would only encourage this more.
  2. It's still heavily skewed toward reputation. 20 out of 40 possible points are earned from reputation (which is floored rather than rounded), so in essence you have the same problem. However, this does decrease the effect of reputation to a certain extent. The main problem here is that the candidate score metric is not detailed enough.
  3. It doesn't take into account a lot of important moderation factors. Among these are Meta participation (you could make an argument for the Quorum and Convention badges fulfilling this, but they're not weighted heavily enough and are fairly easy to earn.), flag counts (an old metric that I think still has merit despite its tendency for gaming), and time spent reviewing in the review queues.

So, while I think candidate scores are a decent metric to be used here, I'm holding out for a better solution.

Overall, I really think that the problem here is with the bottleneck process itself. Simply stripping candidates before the primaries is a bit reckless; there has to be a better way to make decisions about who should be given the chance to run.

  • 2
    You could resolve #1 by using the candidate score of nominees at the time that the nomination opened, maybe, or even at the time the upcoming election was announced. – Jason C Apr 7 '15 at 7:26
  • 1
    Yeah, this doesn't go far enough. Would this user run for moderator, he would have the same score as you (21/40), but having only 5 posts with 26k rep. This is not fair. – Artjom B. Apr 7 '15 at 8:33
14

Reading the question reminded me of something Joel wrote about interviewing:

On the other hand, if you reject a good candidate, I mean, I guess in some existential sense an injustice has been done, but, hey, if they're so smart, don't worry, they'll get lots of good job offers.

There will be more elections in the future and there are plenty of other sites where people might be appointed or elected as a diamond moderator. If our stupid criteria eliminates some ideal candidates, that's unfortunate, but not tragic.

The nomination period is actually a lot more like sorting resumes, though. During the primary phase we will be asking tens of thousands of people on Stack Overflow to have an opinion on 30 candidates. That reminds me of the exercise I used to do everytime I had a meeting at the previous job: estimating how much the meeting costs. If each nomination take, say, 1 minute to read and all 21,571 people who voted have one more nomination to consider, that works out to 15 days of programmer productivity. Yeah, it's a meaningless calculation in oh so many ways, but my point is that we owe it to the voters to narrow down the list somehow.

That said, I like your suggestion of swapping reputation for candidate score is a good one. Reputation alone mostly captures a candidate's investment in the site. The other components get at a candidate's aptitude and enthusiasm for tasks that are more typical for a moderator.

  • That's fair; by no means did I mean to suggest that reputation should be stripped out of consideration entirely. It's an important metric that shows site engagement, and it's certainly valuable in terms of demonstrating investment in the site. I just think it shouldn't be the sole factor (or valued so heavily, as it is with candidate score). It's definitely something to think about, as Tim suggests. – AstroCB Apr 7 '15 at 19:12
9

I totally agree reputation isn't a good factor to filter on. Some users are here for a long time, accumulating reputation from old posts that got a lot of votes. That's totally fair, from a 'how useful was this post' point of view, but doesn't say anything on the moderator capabilities.

Some factors I had like to see:

  • Give extra weight to the gold badges related to moderation, like Marshal, Steward, etc. Maybe on SO and other larger sites, make this the minimum required badges before allowed into the nomination, or give them extra weight in the candidate score.
  • Give a 'meta score'. Since we don't have rep on meta, it is hard to see how the user's opinion on the community is appreciated by the community. We could show the number of Nice Question or Nice Answer badges, or a total score as if there was rep on meta.
  • What about reputation per time? – Jason C Apr 7 '15 at 7:33
  • You mean 'this year'? – Patrick Hofman Apr 7 '15 at 7:34
  • 2
    Some type of "meta score" would definitely be useful. – AstroCB Apr 7 '15 at 7:34
  • No I mean average rate since joining, with a minimum membership time requirement. – Jason C Apr 7 '15 at 7:34
  • @JasonC: Not totally fair too. If you get 20 rep a day from old posts, those are 'free' too. I got your point though, need to find a good way to calculate this. Maybe use the reputation on this year, excluding old posts. – Patrick Hofman Apr 7 '15 at 7:35
5
+250

There are already a number of heuristics in place to make sure people who aren't prepared can't be nominated. I'm not eligible - I don't have all the necessary badges.

Once that's taken care of, you have great people who will be moderators.

Here's where reputation comes in: In every past election the votes have always skewed heavily towards those with higher reputation even after reputation was hidden in the elections.

While we might like to bandy about the idea that reputation doesn't mean good moderation, the reality is that people vote more heavily for higher reputation users.

So this heuristic is in place not because reputation is an indicator of moderator success, but is an indicator of election success. Those with the lowest reputation in the cohort simply do not get elected. So it's best to weed them out early.

32 people is plenty of people to vote on. They could be chosen randomly from the nomination for all it really matters - but it makes sense to pre-select the strongest candidates, and voting history shows us how to accomplish that goal without inserting yet another voting step - which would simply mimic the same process.

Side commentary

Look, let's attack this whole "reputation doesn't mean a hill of beans" thing head on.

The reality is that those with high reputation have it because they enjoy being here, and they have the skills to interact favorably with the rest of the community.

Mostly, the writing skills, and the ability to understand another person, then help them resolve their problem.

In other words, their reputation is actually a better indicator of how helpful they are and how much time they spend here than of how expert they are.

Maybe:

expertise != moderation.

I believe that there's a strong argument, however, to be made for:

high reputation == helpfulness
high reputation == time

Both things that are required for moderation.

I don't think choosing by reputation to be the evil spoken of in the OP.

See that bolded part above? I suspect that the same helpful, congenial writing style that attracts so many upvotes towards reputation is evident in the election writings. Whether it's charisma, or obvious "good fit" with the established community "style" is irrelevant. Reputation, at least indirectly, measures how well this person interacts with and fits in with the rest of the community, which is more important to moderation than a lot of other, poorer heuristics.

3

The problem is two-fold: you don't want to rule out good candidates who are below the rep threshold, and you don't want to have so many candidates that the community can't reasonably evaluate the field. But you might be able to still have both by changing the process. (I am loosely adapting what follows from large martial-arts tournaments I used to participate in.)

If you have, say, 50 candidates, it's not realistic to have everybody vote on all 50. You need an elimination mechanism. This is why there's a primary to winnow 30 down to 10, but your problem is that you need to winnow a larger field down to 30. But you can't do it like the primary, because then everybody has to review every candidate and you may as well just have a huge primary, which is suboptimal.

Instead, allow candidates who would be kept out of the top 30 (by reputation) to challenge in. A candidate may choose one candidate (who is in the top 30 and who has not already been challenged) to directly compete with for the slot. A strong candidate who isn't high-rep gets a chance to compete, but everybody doesn't have to look at all candidates.

This should be a quick, lightweight, head-to-head runoff; you don't want to delay the rest of the election too much and you don't want to open it up for a lot of extra campaigning. You should've made your case already. This is just a first cut, but perhaps something like this:

  • This phase runs for 24 hours, commencing with the close of nominations.
  • A candidate who is "below the line" can click a "challenge" button on a candidate who is above the line -- first come first served; the button goes away once it's been used.
  • The election page displays (perhaps in a new tab) the nomination posts (with comments) for both together, with a vote button for each. (All challenges would be shown on this page.)
  • Anybody who's eligible to vote can click on exactly one of the two vote buttons. Alternate idea: A single voter can vote in only one challenge, to contain the "too many candidates to review" problem (pick the race you most care about). Either way, some subset of the community votes between the two.
  • Either candidate can withdraw during this phase, ceding the slot to the other.
  • 24 hours after nominations closed, the winners of these challenges and the unchallenged remainder of the top 30 proceed to the primary. The primary can be shortened by one day to keep the overall election cycle the same length; think of the challenge round as "pre-primary".

Yes I know that this breaks down if you have more than 60 candidates. Gotta draw the line somewhere; if you have more than that, some of them just aren't going to get a chance.

I don't have a horse in this race. I just enjoy trying to solve systems problems like this.

  • I don't like the idea of challenging one person. That seems a little unfair on a) that person (especially if they are close to getting in) and b) it would be better if it was a random assignment, or maybe "pick the top two out of 5" type thing. I never have liked head to heads, for this reason: i.stack.imgur.com/pUIl3.png – Tim Apr 7 '15 at 15:43
  • @Tim it has to be implementable and not too much strain on the users. Your "but I'm better than (winner of other bracket)" problem is mitigated by the fact that you get to pick who you challenge; if #31 thinks he's better than #29 -- or #3 for that matter, 'cause 3 has lots of legacy rep but hasn't been around much lately -- then he can challenge that one. – Monica Cellio Apr 7 '15 at 15:49
  • My image was meant to represent a "score" - i.e. 6 got kicked out but is actually better than 4, which got through. You see the same thing on TV competitions, like master chef... – Tim Apr 7 '15 at 16:03
  • 2
    @Tim but you're talking about random-seat full-bracket tournaments. Specific challenges have different dynamics. This is to allow people to pursue the claim "I am a better candidate than this person who's in by default". If you can't make that claim about any of the top 30 candidates then this doesn't apply -- but in that case you weren't going to win anyway, so no loss. – Monica Cellio Apr 7 '15 at 17:08
  • Yes, fair point. If it is always top 30 vs bottom X then this issue shouldn't apply. – Tim Apr 7 '15 at 17:09
  • Yes, this is only about letting people who would have been excluded by rep challenge in. It's not about #2 challenging #3. – Monica Cellio Apr 7 '15 at 17:12
  • Yeah, I was forgetting :) – Tim Apr 7 '15 at 17:13
3

One thing that might be good in the mix (both here and in other areas) is some measurement of successful moderation actions, ideas including, among other things:

  • % of edit reviews where your review agreed with the final decision.
  • % of flags helpful.
  • % of close votes on questions that were actually closed (and perhaps not reopened).
  • % of delete votes on questions that were deleted (and perhaps not undeleted).
  • Etc.

I'm not sure how the various components would be weighted among each other and along with reputation (if that's still included), badges, etc. but maybe they are at least somewhat decent indicators of how various moderator actions reflect the community's norms.

Some minimum number of each stat considered here would have to be required, though, to reduce noise.

Also, perhaps only recent (say, within the last year) activity could be considered, as people tend to learn and improve over time (and also this filters out, say, somebody who was very active a few years ago then disappeared).

I don't necessarily think this is the best idea (esp. see Servy's comments below) but just throwing something out there to think about.

  • 2
    For me the key thing to look at for a potential mod is their ability to handle difficult, controversial, or highly ambiguous moderation issues. Dealing with situations where there isn't a clear correct answer that will have consensus among everyone that looks at it, or situations where what is seen at first glance isn't actually the correct course of action. While a particularly high deviation from other users would be a red flag, a particularly low deviation would be equally problematic. It'd be a sign that the user is just farming easy reviews and not actually tackling hard problems. – Servy Apr 7 '15 at 15:11
  • 1
    (Note that just tackling the easy review actions isn't wrong, or bad, it's just not a quality one would want to see in a mod. Mods are there to handle the situations so difficult the community couldn't solve it on their own. Someone with very high percents of these metrics is likely going to be better off helping moderate as a community member than as a diamond.) – Servy Apr 7 '15 at 15:13
  • @Servy Agreed. I'm just brainstorming. It would be a fun data mining project to try and find some pattern that was correlated with handling of controversial decisions, if one even exists. It's too bad most of the data isn't publicly available. – Jason C Apr 7 '15 at 16:15
1

While reputation most certainly does not work in all situations, I believe that it's appropriate to use reputation to sort candidates at such an early stage.

The purpose of the nomination phase is so that we don't need to be looking at so many users to vote from during the primary phase. When there are so many candidates available to choose from, the signal to noise ratio becomes even worse, and it becomes much harder to choose a candidate that will actually be suitable for the position.

Especially given that there are still 30 candidates to choose from during the primary phase, I doubt that there would not be one person from there that would not be willing and capable of handling the responsibilities of being a moderator on one of the world's largest Q&A site.

Most other metrics to measure a user's participation on the site can be gamed without other users being able to easily audit the user's activities. I'd even say that a reputation cap of 5k would be appropriate on a site like Stack Overflow, given that you only gain close voting privileges at 3k and it takes time to understand how it works.

  • ...and it takes time to understand how it works. You've hit on my main point right there. For some people, the time between 3k and 5k could be 10 days, and for others, it could be 2 years. Reputation is not a reliable factor for determining how experienced someone is with moderation or how well they use moderation tools, and this method of bottlenecking won't be fair until it is based off of something else (or something more than just) that metric. – AstroCB Apr 7 '15 at 7:21
  • 1
    @AstroCB: like you, I certainly don't have that much reputation on Stack Overflow. However, at some point where there are just too many candidates to choose from, there has to be some way of lowering them down to an acceptable level. Most other options such as review badges and close vote counts don't really help to gauge whether a user would make a good moderator or not. – Qantas 94 Heavy Apr 7 '15 at 10:48
1

The biggest problem with using reputation as a standard for candidates is that it doesn't actually represent the reputation of the candidate. Your reputation is what others think and feel about you; the SO reputation is actually a record of your activity on the site.

The closest thing to true reputation is the bounty system although it would be great if you could freely assign bounty to people that have been helpful in editing your posts or providing helpful comments in the manner of a tip. Of course there is the risk of gaming the system but that's just part of the site...

The current reputation system is also slightly skewed towards long-time members, you could have achieved 40/40 for the purposes of the election it would be helpful to see the average activity over the last week/month/year so that we could better gauge the recent activity of a candidate.

I feel the combination of reputation and activity would be the most helpful means of identifying the minimum bar for candidate selection.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .