37

During a Moderator Election we are provided with details about each candidate, including their meta participation (questions and answers) and a total count of helpful flags. These details help to see the activity each candidate has on the site.

One of the biggest things in the moderators job is handling flags, including comment, spam, custom, etc. Currently, the helpful flag count shows all of these in a single total.

Using my SO nomination as an example:

enter image description here

It shows the total flags marked helpful, but doesn't really offer any details as to what I flagged. It's possible that 20k of those flags were on comments.

Comment clean-up is great because it is helping to maintain the site, but flagging comments is not the same as making a decision on whether or not a post is "Not An Answer" or should be removed for some other reason.

After discussing this others, instead of providing a full breakdown of all flags, I suggest we remove the comment flags from the helpful flag count. Stripping out the comment flags the result would be:

enter image description here

While I initially thought that a full breakdown of flags stats would be beneficial, I've realized that providing a full breakdown of all flags could be far too many numbers to digest. Comments are relatively easy to flag and are a great way to increase flag stats, there are even robots doing it. But flagging comments like "Thanks", etc. takes minimal effort and doesn't show the same judgment that might be needed to moderate other content.

  • I'd suggest a subtle icon that suggests there is a breakdown, visible upon hovering over said icon. Otherwise it looks like we're shoving too many numbers into a small box. – animuson Dec 30 '14 at 21:17
  • @animuson I'm good with that, I'm not a designer - anything is better than what we've got right now which is bupkis. – Taryn Dec 30 '14 at 21:19
  • 12
    Do you really care about the feature-request, or do you just want an opportunity to show off how much you've flagged? :P – Doorknob Dec 30 '14 at 21:46
  • 5
    @Door Heh. I flagged many more posts but I don't boast at least. – nicael Dec 30 '14 at 21:48
  • Only your blue feet and god know :D – nicael Dec 30 '14 at 21:51
  • What prompted this post? Is there an election coming up? – CRABOLO Dec 31 '14 at 2:42
  • @Sompuperoo I'm not aware of any upcoming elections, but bots are being used to flag comments, which increases the helpful flag count tremendously. Flagging comments is not hard, there are tons of them. But flagging content (questions/answers) that need to be removed is much more difficult to do and get right. – Taryn Dec 31 '14 at 11:39
  • 4
    Just an idea: Shouldn't the stats simply get frozen at the beginning of the election? – yo' Jan 15 '15 at 19:55
53

Update: the system described below is live for all past and future elections

Big thanks to Ol' Slim and his merry band of stats-obsessed voters over on Mathematics Stack Exchange for coming up with this brilliantly simple idea. Per venerable tradition, details on how each line-item contributes to the total score are displayed via title-tips - hover your cursor over the line to see which badges are involved.


Original answer: rationale for not implementing this as-requested

The big issue I have with this is that we're making a value judgement on something that's... Pretty iffy all around. We want folks to flag truly problematic comments, just as we want them to flag truly problematic posts. As you note, it's easy enough to "farm" flag counts by flagging worthless (but not overtly problematic) comments, but it's also easy enough (on larger sites at least) to do the same thing with VLQ or Not Answer flags. Heck, there are folks who've built up respectable flag counts via tools that quickly identify spam - do we want to strip this down to counting just Other flags?

I think we're throwing the baby out with the bathwater here. The real problem is that a raw count doesn't say much... In fact, that was my rationale for not including a count for reviews - but this suffers from the same problem, in that most folks won't do what I wanted there either (look at the actual reviews, form an opinion on the suitability of the candidate from them).

An intentionally hobbled score

Some Math users came up with an interesting idea during their last election:

... it resulted in the "Citizenship Score" query, which ranks users on the scale from 1 to 40 as follows:

  • 1 point for each 1K reputation, up to the maximum of 20 points. Users below 1K rep are not included.
  • 1 point for each of the badges Civic Duty, Cleanup, Constituent, Convention, Copy Editor, Deputy, Electorate, Enthusiast, Explainer, Investor, Marshal, Organizer, Quorum, Refiner, Reviewer, Sportsmanship, Steward, Strunk & White, Tag Editor, Yearling. (Each badge is counted only once.)

The purpose of counting these badges is to approximately quantify the contributions that do not result in reputation (edits, votes, meta posts, tagging, reviewing) as well as the level of experience with the site.

I like this idea a lot, because it makes it hard for someone to dominate the list with one statistic (rep, flags) if they haven't done much of anything else. Also, it has the potential to scale a lot better than the current metrics (and restrictions such as those on Stack Overflow). My concerns regarding such a score applied to the general population don't apply to elections, since we're not trying to rank, say, askers according to their flagging prowess - we're explicitly looking for well-rounded - exemplary, even - citizens.

Flag counts aren't public data, so that didn't figure into Math's score - but there's no reason we couldn't include them if we wanted to. We could even cap the influence of each type of flag on the overall score, offering a small benefit to folks who don't fixate on one type of problem. We could lump close votes and close flags together in this, and offer a breakdown for those interested in seeing what a given candidate's "baseball card stats" look like.

Most crucially, this score could replace reputation in the list of candidates, potentially offsetting the natural tendency of some voters to vote according to reputation and nothing else. Imagine if the nominations were signed like this:

This idea still needs some fleshing out, but I strongly believe we'll want to implement something like it for elections in the near future. Particularly with recent changes to flagging associated with Review, the utility of a raw Flags Raised stat has dropped considerably - time to re-think the whole plan.

  • 6
    I would absolutely love to see citizenship score implemented in elections, this sounds perfect. – Undo Jan 15 '15 at 19:06
  • 5
    I like this idea. Right now, the current metrics we are given aren't detailed enough or could easily be skewed prior to or even during an election (by mass flagging). Coming up with some other stat/score, whatever we call it, would be extremely helpful in reviewing the moderator candidates. – Taryn Jan 15 '15 at 19:17
  • 4
    The problem remains the same. You try to express different mechanics into one number that is presented for electees to judge. Reputation and flag count are indicators every user is familiar with. Citizenship not, that number only pops in an election. I think you have to be careful that an election isn't reduced to db-query to select the top 3 users with the highest citizenship number that indicated in their profile that they want to be a moderator. Don't loose the human aspect out-of-sight, don't focus to much on a number that fits/feels right.... – rene Jan 15 '15 at 19:25
  • 1
    Ideally the score should be capped in a way that most of the good candidate have near-max scores. That would alleviate the issue with users voting solely according to the score to some extent while still clearly separating candidates that have made some effort moderating the site from those that haven't – Mad Scientist Jan 15 '15 at 19:34
  • 3
    I'm not a fan of using badges for this, though. The badge requirements are far too coarse-grained, I'd use the number of edits, reviews, flags and meta posts directly. – Mad Scientist Jan 15 '15 at 19:36
  • @rene Is it that way now? I've passed over high-rep high-count candidates in elections because they've written a crappy nomination post. The score matters a little, but I don't think it's the main determinant of who wins an election. – Troyen Jan 15 '15 at 21:21
  • 1
    @Troyen passing over a candidate can be based on a lot of things. Your reason feels as a better aproach. I only try to express that the statistics are not or should not be the key decision maker in an election. And I'm not a fan of having a number that isn't in our genes. We can't do without some stats and revealing stuff that is normally hidden seems ok but another stat? I dunno... – rene Jan 15 '15 at 21:40
  • 5
    You may be overestimating how familiar voters are with flag counts, @rene - last SO election, only 65% of voters had even a single helpful flag; the remaining would've never seen a flag count outside of an election. – Shog9 Jan 15 '15 at 22:53
  • I'd add that the flag counts were useful to me in that election -- one candidate had less than 40% of their flags marked useful; it would be a pity to lose that indicator altogether – Glen_b Apr 3 '15 at 3:15
  • 1
    I'll wager the candidate you're thinking of probably doesn't look too shiny in the new system either, @Glen_b. – Shog9 Apr 3 '15 at 3:25
  • 1
    Thanks Shog9; not too shiny ... but the flags helped me to make a choice between that one and another candidate when trying to express a third preference -- the new information doesn't distinguish between them at all, whereas the very low helpful flag rate was a sufficient red flag to help distinguish the two. Judging from comments it was useful to more people than me. – Glen_b Apr 3 '15 at 3:46
  • 1
    Fair point; we'll have to see how this plays out in practice - information density could be increased slightly, but we're pushing it. – Shog9 Apr 3 '15 at 17:42
  • @Glen_b The count of helpful flags will soon be visible to everyone even outside of elections; it's prominently shown in the redesigned user profile. (As an aside, the Stats elections simply did not have 3 viable candidates; making a distinction between 3 and 4 wasn't consequential.) – user259867 Apr 4 '15 at 2:24
  • @pizza That's true, but I don't think the majority of voters will go beyond the nomination blurb to check out a user's profile. For this information to be useful, it should be in there. – AstroCB Apr 7 '15 at 5:59
  • 3
    And @Shog9, on that note, I think the information density of these blurbs should be high. If a user is nominating him/herself for this position, they should be willing to display their laundry. If they're not proud of their actions and stats, they shouldn't be running. – AstroCB Apr 7 '15 at 6:01
24

There's another historical reason why this makes sense. Before June of 2013, comment flags didn't count as "helpful" flags in the statistics. They didn't affect the numbers one way or another. Anyone who cast a comment flag before that point doesn't get recognized in the current flag counting scheme, if I remember correctly.

Comment flags were made to count as full helpful flags when cleared in part to give moderators an incentive to handle them (flag handling stats for each of us are shown in an internal leaderboard). This also ended up encouraging a lot more comment flagging by members of the community, because it's pretty easy to find comments to flag. Even robots can do it.

Let's be honest, most voters (at least during the Stack Overflow elections I've watched) tend to vote based on numbers they see, not moderator statements or Town Hall transcripts or anything else. For otherwise equal candidates, people vote for those with the highest reputation and those with the highest helpful flag count. Clarifying this number will matter to voters and I think emphasize those most likely to be able to handle our workload.

One other thing I'd like to see: the numbers displayed in the election listing should be a snapshot as of the start of the election. I've observed too many candidates on Stack Overflow attempt to game this by mass-flagging things during the election in order to boost their numbers. Locking this to when the election starts would prevent this gaming.

  • 11
    Agree, especially with that last paragraph. I've seen a few users who have run for election and become extremely active during the election phases, then kind of dropped back off the face of the planet afterwards. – animuson Dec 31 '14 at 15:59
5

(Remark: I realized that Brad's answer suggests this, too. I think it's better to keep this answer since it lists more details on the reasoning.)

IMHO the simplest solution would be: Freeze all the stats at the very beginning of the election.

If the user hasn't contributed before the election was announced and starts contributing during the election, what are the chances that he'll continue once it's over? I doubt they are high. This solution simply doesn't allow manipulation with the stats, no matter what.

Yes, people still can be flagging lots of comments to boost the helpful flags stat, but they'd have to do it before the election starts, and I'm not sure if there's anything wrong with this, after all. The issue really seems to be in people boosting the stat once they decide to get in.

I can see a minor issue here only with relatively new users participating in the election, however, they always have the cover text to explain how things are.

  • Yes, this would be helpful and Brad suggests this in his answer. – Taryn Jan 16 '15 at 11:10
  • @bluefeet Oh, I didn't notice he suggests that so directly (I went only briefly through the other answers). Should I delete this? – yo' Jan 16 '15 at 11:13
  • I'll leave that up to you. :) – Taryn Jan 16 '15 at 11:13
  • Ok, I'll keep it since I give a bit more details on the arguments. – yo' Jan 16 '15 at 11:19
3

If any flags were to be excluded, I think only the obsolete comment flags should be. Offensive flags are always important, and custom flags take enough effort that if they're accepted then they should be counted. What about non constructive or too chatty? Maybe it depends on the mods, but I'd feel that trying to boost your numbers through those flags wouldn't result in many being accepted, so the ones which are should count too. Obsolete comments are the only kind of comment flags which can be easily found, reliably accepted and result in minimal benefit to the community.

  • 1
    I understand your point to excluding obsolete flags but everyone's definition of obsolete is a bit different. Would that be "Thank you" comments or "I've edited the post" comments? Everyone uses a different definition of not constructive, obsolete, and too chatty. Possible we exclude those and not the offensive or custom comment flags. – Taryn Jan 16 '15 at 11:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .