Exclude comment flags from the helpful flag count during moderator election

Question

During a Moderator Election we are provided with details about each candidate, including their meta participation (questions and answers) and a total count of helpful flags. These details help to see the activity each candidate has on the site.

One of the biggest things in the moderators job is handling flags, including comment, spam, custom, etc. Currently, the helpful flag count shows all of these in a single total.

Using my SO nomination as an example:

It shows the total flags marked helpful, but doesn't really offer any details as to what I flagged. It's possible that 20k of those flags were on comments.

Comment clean-up is great because it is helping to maintain the site, but flagging comments is not the same as making a decision on whether or not a post is "Not An Answer" or should be removed for some other reason.

After discussing this others, instead of providing a full breakdown of all flags, I suggest we remove the comment flags from the helpful flag count. Stripping out the comment flags the result would be:

While I initially thought that a full breakdown of flags stats would be beneficial, I've realized that providing a full breakdown of all flags could be far too many numbers to digest. Comments are relatively easy to flag and are a great way to increase flag stats, there are even robots doing it. But flagging comments like "Thanks", etc. takes minimal effort and doesn't show the same judgment that might be needed to moderate other content.

Answer 1

Update: the system described below is live for all past and future elections

Big thanks to Ol' Slim and his merry band of stats-obsessed voters over on Mathematics Stack Exchange for coming up with this brilliantly simple idea. Per venerable tradition, details on how each line-item contributes to the total score are displayed via title-tips - hover your cursor over the line to see which badges are involved.

---

Original answer: rationale for not implementing this as-requested

The big issue I have with this is that we're making a value judgement on something that's... Pretty iffy all around. We want folks to flag truly problematic comments, just as we want them to flag truly problematic posts. As you note, it's easy enough to "farm" flag counts by flagging worthless (but not overtly problematic) comments, but it's also easy enough (on larger sites at least) to do the same thing with VLQ or Not Answer flags. Heck, there are folks who've built up respectable flag counts via tools that quickly identify spam - do we want to strip this down to counting just Other flags?

I think we're throwing the baby out with the bathwater here. The real problem is that a raw count doesn't say much... In fact, that was my rationale for not including a count for reviews - but this suffers from the same problem, in that most folks won't do what I wanted there either (look at the actual reviews, form an opinion on the suitability of the candidate from them).

An intentionally hobbled score

Some Math users came up with an interesting idea during their last election:

... it resulted in the "Citizenship Score" query, which ranks users on the scale from 1 to 40 as follows:

• 1 point for each 1K reputation, up to the maximum of 20 points. Users below 1K rep are not included.

• 1 point for each of the badges Civic Duty, Cleanup, Constituent, Convention, Copy Editor, Deputy, Electorate, Enthusiast, Explainer, Investor, Marshal, Organizer, Quorum, Refiner, Reviewer, Sportsmanship, Steward, Strunk & White, Tag Editor, Yearling. (Each badge is counted only once.)

The purpose of counting these badges is to approximately quantify the contributions that do not result in reputation (edits, votes, meta posts, tagging, reviewing) as well as the level of experience with the site.

I like this idea a lot, because it makes it hard for someone to dominate the list with one statistic (rep, flags) if they haven't done much of anything else. Also, it has the potential to scale a lot better than the current metrics (and restrictions such as those on Stack Overflow). My concerns regarding such a score applied to the general population don't apply to elections, since we're not trying to rank, say, askers according to their flagging prowess - we're explicitly looking for well-rounded - exemplary, even - citizens.

Flag counts aren't public data, so that didn't figure into Math's score - but there's no reason we couldn't include them if we wanted to. We could even cap the influence of each type of flag on the overall score, offering a small benefit to folks who don't fixate on one type of problem. We could lump close votes and close flags together in this, and offer a breakdown for those interested in seeing what a given candidate's "baseball card stats" look like.

Most crucially, this score could replace reputation in the list of candidates, potentially offsetting the natural tendency of some voters to vote according to reputation and nothing else. Imagine if the nominations were signed like this:

This idea still needs some fleshing out, but I strongly believe we'll want to implement something like it for elections in the near future. Particularly with recent changes to flagging associated with Review, the utility of a raw Flags Raised stat has dropped considerably - time to re-think the whole plan.

Answer 2

There's another historical reason why this makes sense. Before June of 2013, comment flags didn't count as "helpful" flags in the statistics. They didn't affect the numbers one way or another. Anyone who cast a comment flag before that point doesn't get recognized in the current flag counting scheme, if I remember correctly.

Comment flags were made to count as full helpful flags when cleared in part to give moderators an incentive to handle them (flag handling stats for each of us are shown in an internal leaderboard). This also ended up encouraging a lot more comment flagging by members of the community, because it's pretty easy to find comments to flag. Even robots can do it.

Let's be honest, most voters (at least during the Stack Overflow elections I've watched) tend to vote based on numbers they see, not moderator statements or Town Hall transcripts or anything else. For otherwise equal candidates, people vote for those with the highest reputation and those with the highest helpful flag count. Clarifying this number will matter to voters and I think emphasize those most likely to be able to handle our workload.

One other thing I'd like to see: the numbers displayed in the election listing should be a snapshot as of the start of the election. I've observed too many candidates on Stack Overflow attempt to game this by mass-flagging things during the election in order to boost their numbers. Locking this to when the election starts would prevent this gaming.

Answer 3

(Remark: I realized that Brad's answer suggests this, too. I think it's better to keep this answer since it lists more details on the reasoning.)

IMHO the simplest solution would be: Freeze all the stats at the very beginning of the election.

If the user hasn't contributed before the election was announced and starts contributing during the election, what are the chances that he'll continue once it's over? I doubt they are high. This solution simply doesn't allow manipulation with the stats, no matter what.

Yes, people still can be flagging lots of comments to boost the helpful flags stat, but they'd have to do it before the election starts, and I'm not sure if there's anything wrong with this, after all. The issue really seems to be in people boosting the stat once they decide to get in.

I can see a minor issue here only with relatively new users participating in the election, however, they always have the cover text to explain how things are.

Answer 4

If any flags were to be excluded, I think only the obsolete comment flags should be. Offensive flags are always important, and custom flags take enough effort that if they're accepted then they should be counted. What about non constructive or too chatty? Maybe it depends on the mods, but I'd feel that trying to boost your numbers through those flags wouldn't result in many being accepted, so the ones which are should count too. Obsolete comments are the only kind of comment flags which can be easily found, reliably accepted and result in minimal benefit to the community.