7

Flag weight is the total number of flags marked as valid helpful minus the number of flags marked as invalid declined for a given user. That number is used…

… well, it's shown on a user's profile, and it's used for a couple of badges, …

… oh, and there's one useful thing: it's used to prioritize Stack Overflow's huge flag queue. But there are better ways to do that.

Now the case for the prosecution: flag weight gets discussed a lot. Most of this discussion isn't helping anyone, it's not making SE a better place and it's wasting moderators' time. Flag weight also gets people to flag for the sake of flag weight, at best wasting moderators' time and at worst harming the site. And recently there's been some friction with moderators — the recent flag dismissal reform.

So please remove the concept of flag weight altogether, as well as the distinction between valid helpful and invalid declined flags. Keep the optional ability for moderators to give feedback — meaningful feedback, as opposed to the single bit of signal that existed until recently. Feedback could be things like “we don't do community wiki for that stuff any more” or “yup, the question is off-topic, but that migration makes no sense” or “you could have cast a close vote”.

UI mock-up

(The “send feedback” button does just that. If the input text is empty, that button is a simple “dismiss”. Closing or deleting the post automatically dismisses without sending feedback, so if you want to send feedback, do it first. UI design is not my forte, so feel free to improve.)

Flag weight is supposed to “deter serial abusers”, but there's no need for such a complex tool, and it doesn't deter all abuses anyway. If a moderator sees a suspicious flag, we have the option to go through the user's flag history (please keep that!) and visually eyeball if that user had a lot of bad flags. Visually eyeballing is fine to detect serial abusers.

N.B. Although this post has been marked , my request has not been implemented. Flag weight and flag validity have not been removed. Flag weight is no longer displayed on profiles, which is an improvement, but flag validity still exists, leading to arguments about rejected flags, which are the only way to provide feedback, but are often perceived negatively.

19
  • 3
    So.. you want to switch out an automatic quality tracking system with a manual one, in order to save moderator's time? That doesn't seem like it would work, in practice.
    – agf
    Sep 18, 2011 at 20:36
  • 2
    @agf It's not automatic, mods still need to manually indicate if a flag is good or bad and what's wrong with it Sep 18, 2011 at 20:37
  • 1
    @agf No, I want to replace the system that requires moderator time for each flag by one that requires moderator time only for suspected abusers. Sep 18, 2011 at 20:39
  • I meant the evaluation of the users status as a "serial abuser", "useful flagger" etc. is automatic, instead of requiring the mod go through the user's flag history.
    – agf
    Sep 18, 2011 at 20:40
  • @agf: I see it as replacing an automatic system that requires constant maintenance and doesn't do anything useful with a manual system that requires infrequent maintenance and does what it's supposed to (help educate mediocre flaggers and quash the serial abusers).
    – Aarobot
    Sep 18, 2011 at 20:41
  • 18
    I understand why you want to remove the actual displaying and the associated badges, but why remove the proritizing? That was the original reason for this, and as far as I can tell, it's working quite well.
    – balpha StaffMod
    Sep 18, 2011 at 20:42
  • I'm all for this except for the prioritizing part as @balpha mentions - that seems to be working quite well
    – Pekka
    Sep 18, 2011 at 20:48
  • 2
    @balpha I want to remove the valid/invalid distinction altogether. Free-form feedback passes the signal much better. The only site where prioritizing flags is useful is SO, and Tim Post has better suggestions for that. Sep 18, 2011 at 20:50
  • 6
    @balpha: I suppose the reason is that this prioritizing is not the best way to prioritize flags. Good flaggers (i.e. people with high flag weight) have both urgent flags (like an account that needs merging, fresh question which needs migration) and non-urgent flags ("not an answer" on posts which are years old, or deletion/close requests on old questions), and it is not right to handle them the same way. Even more if there are ways to inflate the flag weight arbitrarily by easy measures (like reflagging what was already flagged), so "good flagger" is not even an objective measure. Sep 18, 2011 at 20:52
  • I wonder if any of the people downvoting this are moderators.
    – Aarobot
    Sep 18, 2011 at 21:10
  • You know how this was inspired by Newgrounds' whistle system? I've been a user there for 7 years and I can tell you it's been equally effective both here and there. Sep 18, 2011 at 21:10
  • @BoltClock Maybe Newgrounds uses the whistle weight for something? On SE it's only used to prioritize the SO queue, and there are better ways to do that. Sep 18, 2011 at 21:13
  • 2
    @RobertHarvey: Why do I wonder how many people are voting who have no skin in the game?
    – Aarobot
    Sep 18, 2011 at 21:52
  • 3
    @Aarobot: Ah, I see. Well, I didn't downvote it, but I don't think tossing the system out is the right solution.
    – user102937
    Sep 18, 2011 at 21:53
  • 4
    I like the flag weight. It motivates me to flag more. I am trying to reach the coveted 750 flag weight (I am at 714 now). I use up all my flags daily. Most are on "Not an answer" type posts (i.e. "Me Too!" or "Can you help me, I am having the same problem?" by 1 rep users). I want to keep it. Overall, we are doing the cleanup work and giving it a priority for the Moderators to handle. This wouldn't be an issue if people would downvote more and comment on bad answers/questions. Sep 21, 2011 at 19:50

6 Answers 6

23

Flag weight should have been linearly capped at 2000 (or some other arbitrary number), instead of the current asymptotic limit at 750. I believe a linearly-capped number would have avoided a lot of the problems we are having with flag weight.

The asymptotic limit disproportionately punishes good flaggers, and it causes cognitive dissonance for me as a moderator, since I have to take the current flag weight of the user into account when considering whether or not to decline a flag.

Consider the following, real-world example (names removed to protect the innocent). Occasionally a user gets a wild hare and decides to flag a bunch of posts en-masse for something. In this particular case it was foul language. The problem is that about half of those posts contain code that either filters bad language, or contains music database information (Parental Advisory: Explicit Lyrics).

I declined all of the flags where the bad words were in code. This particular user had a flag weight of about 120. When I had completed servicing his flags, he had a net gain in flag weight, in spite of the multiple declines. Had he had a flag weight of 700 or above, he would have incurred a substantial loss in flag weight.

This problem is exacerbated the closer you get to 749 and that coveted Marshal badge. I don't know how many flags it takes to recover a 10 point weight loss in the 750 range, but it's a lot.

I have a flag weight in the 610 range on Programmers.SE, and I don't expect to ever get the Marshal badge over there. There's not enough volume of posts, for one thing, and the moderators over there can be a bit... erm, fickle with their flag dismissals. :P

22
  • 11
    Your jab at us "fickle" Programmers.SE moderators highlights another seriously broken aspect of the flag weight system: when high flag weight users get a declined flag, they tend to immediately assume it's the moderators who are in error, not themselves. When someone flags as much as a high flag-weight user does, there's no expectation that they're going to bat a thousand: otherwise, we might as well just make flags binding for users over 500 flag weight.
    – user149432
    Sep 18, 2011 at 22:13
  • 1
    @Mark: Ah, I knew I'd get someone's attention over there. :D
    – user102937
    Sep 18, 2011 at 22:18
  • 4
    Seriously, though, someone with 740 flag weight gaming for the Marshal badget wouldn't be flagging something unless they were pretty sure they were correct.
    – user102937
    Sep 18, 2011 at 22:23
  • 4
    So would capping it linearly instead of asymptotically.
    – user102937
    Sep 18, 2011 at 22:27
  • 3
    @RobertHarvey That's why we have moderators to tell them when they're mistaken or incorrect. Somewhere between 100 flag weight and 740 flag weight the way people perceive flags goes from "this is a learning experience" to "I'm entitled to this badge, and you're wrong for denying it from me" but for moderators, we're still acting on the former.
    – user149432
    Sep 18, 2011 at 22:31
  • 3
    @Mark: That's a bit cynical, don't you think?
    – user102937
    Sep 18, 2011 at 22:33
  • 3
    @RobertHarvey It's what you're suggesting: that at 610 or 740 flag weight, users think they know better than the moderators. The other answers here talk about how they use flag weight to know if they got it right: you're suggesting that after a certain point, flag weight is used by users to determine if the moderators got it right, contrary to its intended usage. That's broken: if after a certain point users are supposed to think that any declined flag is a flag handled incorrectly, we might as well just make them moderators and be done with it.
    – user149432
    Sep 18, 2011 at 23:04
  • 1
    @Mark: Hmm. All I'm saying is that someone who flags regularly (and therefore is of value to the community in that regard) should not be punished more than someone who rarely flags. The flag weight of any given individual doesn't have anything to do with the veracity of moderator decisions in my view, nor should it.
    – user102937
    Sep 18, 2011 at 23:21
  • 13
    @Mark: I claim that it's the distorted scoring system of high flag weight that causes this, not the high flag weight itself. People would not complain at all about a declined flag if they could recover it with one good flag instead of 30.
    – user102937
    Sep 18, 2011 at 23:32
  • 2
    @RobertHarvey What's up with this line? "since I have to take the current flag weight of the user into account when considering whether or not to decline a flag." I have never considered that information when judging a flag. A flag is judged on its merits.
    – jcolebrand
    Sep 19, 2011 at 1:45
  • 1
    @jcolebrand: Read more carefully. You're taking that line out of context; read my entire post.
    – user102937
    Sep 19, 2011 at 2:38
  • 1
    I did read your entire post, and you kept going back to giving worth to the existing flag weight. That's not how flags are to be interpreted. If your entire post ignored existing flag weight, maybe, but you're not. You're implying that existing flag weight matters for anything other than queue ordering. As a mod, I never consider flag weight, ever.
    – jcolebrand
    Sep 19, 2011 at 4:17
  • 8
    @jcolebrand: I'm happy for you. I simply would prefer that the weighting system didn't penalize great flaggers more than average ones.
    – user102937
    Sep 19, 2011 at 4:32
  • 4
    Think in the big picture, then. If you're an excellent flagger that reaches 740 flag weight, then yes, it's going to take a lot of good flags to overcome the penalty of a bad flag. But if you're an excellent flagger, isn't that a given that you'll be posting out a lot of good flags? The way the asymptote works, if you actually have a failure rate, then you'll settle down somewhere in the middle as your failures balance your growth. That way, the people who flag better consistently end up higher.
    – Grace Note StaffMod
    Sep 21, 2011 at 18:46
  • 2
    @Robert A disproportionate effect that will be gone pretty swiftly if you're still a good flagger. The only real impact comes if you're getting a lot of flags dismissed, and that's a sign for some talking to in the first place, whether the user initially had 740 or 100 flag weight.
    – Grace Note StaffMod
    Sep 21, 2011 at 19:00
26

While I'm the kind of guy that would flag anyway, I find myself pushing a bit more to flag, going through the review list when I have a free minute, because there is a badge at the end.

I have no idea why it works this way for me, but I've definitely noticed that it does, therefore I think that while the flag weight concept could be modified, it should probably stay in there so that the process can be incentivized.

2
  • 7
    I don't flag for the weight, and considering how many times I've read “I have 740 flag weight and now I'm afraid of flagging borderline cases in case I get set back”, I don't think flag weight is encouraging good flaggers to keep flagging. Sep 18, 2011 at 20:52
  • 2
    Different Strokes. Even though I'm addicted to editing, I find I push to do more on the sites that I still need the badges on. Sep 18, 2011 at 20:53
12

From a user's perspective:

Although it is probably trivial, I enjoy monitoring my flags and trying to keep a high flag weight. Seeing my flag weight and number of valid/invalid flags is the only feedback I have so I can tell if I'm flagging inappropriately.

I have a flag weight of 650 and I think at least 90% of my flags are "not an answer". I guarantee that without knowing my "flag score" or any feedback from the people who process the flags, I would be flagging a lot more posts inappropriately as "not an answer", wasting more moderator time. I've learned from a few invalid flags what the "not an answer" flag is really for.

I know people like to ask about their invalid flags on meta and what not, and sometimes it's annoying, but I'm not sure what's wrong with that. It's good that people are interested in learning when and when not to flag, open to discussing what is appropriate content for SO, and I've learned a good deal from reading those posts.

I think the text change from "invalid" to "declined" will help curb some of that unrest. Although it's just semantics, it does make a difference. Regardless of what the actual intent is, this is how I, personally, interpret the language:

  • Invalid - Your flag is wrong! (No it's not!!! Time to bring this to meta!)
  • Declined - Thanks, but no thanks. (ah, ok - whatever)

"Declined" makes me feel a bit better about it than "invalid". Anyways, change the way feedback is displayed, sure: but don't take it away entirely. I don't need a personal message from a moderator, just a yes/no or a checkmark or something is fine.

I'd imagine a lot of other users would be flagging with a lot less discretion if there was no feedback on how the flag was handled. I know I would be.

5
  • 2
    That's actually exactly the opposite of what the definitions are supposed to mean. "Declined" does not mean "we didn't take action", it means the flag was not useful. Helpful flags are still, well, helpful, even if no moderator action is or was necessary. Thank you for demonstrating how the slippery slope of flag weight as a parallel reputation system is just adding more confusion.
    – Aarobot
    Sep 18, 2011 at 22:11
  • Thanks Aarobot - edited to clarify what I meant, so as not to spread misinformation. I understand that action may or may not be taken on an invalid or valid flag. That bit is really an aside anyways.
    – user159834
    Sep 18, 2011 at 22:15
  • What I'm proposing is to get rid of valid/invalid and instead provide meaningful feedback. What's your objection? Sep 18, 2011 at 22:26
  • 3
    Well, in your post you say: remove the ... distinction between valid and invalid flags. Keep the **optional** ability for moderators to give feedback. My objection is to removing the concept of valid/invalid flags. Sometimes a personal message is appropriate, but in most cases valid/invalid is all the feedback users will need. The way I interpret that part of your suggestion, you are saying that I could have any number of processed flags with no feedback in either direction, is that correct?
    – user159834
    Sep 18, 2011 at 22:34
  • @gilles you'll note that our guidance is to err on the side of dismissing as helpful unless you have a strong belief that the flag is utterly incorrect. I find that the OVERWHELMINGLY VAST majority of flags I handle on SO are indeed helpful, and have been since we started. Very few people flag because they're bored or malicious, and even misunderstood flags are symptomatic of deeper problems on the post -- where there is smoke there is fire. Generally if I fix a problem on a post, and it was flagged, that was helpful regardless of what the flag said. Sep 29, 2011 at 8:33
8

You could make the same (incorrect) argument about reputation. Why show reputation to users? What's the point? All the angst over voting and reputation changes!

And yet, time and time again, the reputation system, while not perfect, produces sane community behavior and results in people getting answers to their questions.

I'd say flag weight is, without a doubt, the #1 thing we've added in the last year that has gone directly toward increasing the bottom line quality on Stack Overflow and in fact the only thing that has given us any hope of scaling Stack Overflow to millions and millions of users without losing all the quality attributes that made Stack Overflow worth visiting in the first place.

We don't just need community elected moderators, we need thousands of mini-moderators, on the site all the time.

This flag system worked for Newgrounds, and it's working for us. But don't take my word for it. I encourage you to look at the actual data and see the enormous volume of horrid, terrible content on Stack Overflow we've been able to remove or fix as a direct result of massive increase in mini-moderator flagging.

Also, Raymond motherflippin' Chen is flagging posts on Stack Overflow (link will be visible to diamond mods only, sorry). When Raymond Chen is flagging posts, it is crystal clear to me that flag weight is working as designed.

So in summary:
RAYMOND CHEN USES FLAGS, YOUR ARGUMENT IS INVALID.

2
  • 10
    "When Raymond Chen is flagging posts, it is crystal clear to me that flag weight is working as designed". Er. Did Raymond Chen say somewhere he was flagging posts because of his desperate desire for a Marshal badge? Sep 29, 2011 at 15:47
  • Is there a way to look at the actual data besides the review tab? Like a giant cemetery data dump? It might be very useful to convince people who still think everyone should get to ask any kind of question
    – Pekka
    Sep 30, 2011 at 6:31
4

Edit: From Jeff's comment below:

It's very simple; if you want people to optimize for a number, make it public.

11
  • 2
    I use up all of my flags daily. If I didn't have that 750 flag limit to motivate me, then I probably could give a crap honestly. It's like a game. Yes, I like making the site better but I need some virtual motivation. For this reason alone, I don't use the 10K Review tab as much because there isn't a "Review" weight or some arbitrary number. I like knowing my Flag Weight, though I wish I could go higher than 750. I am at 714 now and have made about 591 successful flags. Sep 21, 2011 at 19:47
  • 7
    you could make the same (incorrect) argument about reputation. Why show reputation to users? I'd say flagging is about the #1 thing in the last 2 years that has gone directly toward increased quality and given us any hope of scaling Stack Overflow without being overwhelmed by millions of typing monkeys. Sep 29, 2011 at 8:21
  • @JeffAtwood People who attempt to game reputation by posting popular poll questions or something can still be voted down and have their questions closed and deleted. On the opposite side of the scale, all flags that aren't outright wrong are supposed to be marked helpful. And as noted, flag weight is just a number whereas rep unlocks privileges and so on. They're not comparable. I don't disagree that flagging itself is immensely useful. You're still not addressing whether visible flag weight is more useful than invisible flag weight and whether the benefits are worth all the crap.
    – user154510
    Sep 29, 2011 at 16:20
  • 1
    @matt what crap? 99% of the flags I see -- and I've handled ~14,000 flags to date on Stack Overflow -- point out problems with posts that need fixing. Often severe problems. Flag weight unlocks more flags as well as badges. It's very simple; if you want people to optimize for a number, make it public. Exactly like reputation score. So if you're going to argue this, you should also be arguing for hiding reputation scores. Sep 29, 2011 at 16:32
  • @JeffAtwood Now there's a real argument. I can't disagree with if you want people to optimize for a number, make it public, nor can I successfully argue that more flags should be declined given your stats. I yield :P
    – user154510
    Sep 29, 2011 at 16:39
  • 1
    well, you needn't yield so unilaterally; perhaps I am not understanding what the problem is. From my perspective, other than being extra work for the moderators, there is no downside to having more flags. The more flags we have, the more low quality content we can remove or fix on our sites. There is definitely a scale issue on Stack Overflow that we've struggled with, but we are using every trick in the book that we know of to help, and adding more every day. For example, we just started outright blocking questions with certain words in the title. Sep 29, 2011 at 16:46
  • @JeffAtwood I'm aware of the scale issues, but I barely participate in SO as a user; I have no first-hand experience of the scale issues. The sites I'm active in are much smaller. So I do have residual concerns, but those are more or less addressed by the other posts here and elsewhere (and more knowledgably than I could address them).
    – user154510
    Sep 29, 2011 at 16:50
  • 1
    @JeffAtwood: In that case, any re-consideration of my request yesterday for more flags? Considering that you have scaling problem :) Sep 29, 2011 at 16:55
  • 1
    @JeffAtwood, at Matthew: Flag weight is not a useful number to optimize for. There may be a useful number to optimize for; linear flag weight (and why not forever-increasing linear flag weight — like reputation) might be better. But the really important thing is allowing moderators to provide feedback; the helpful/declined boolean isn't very useful compared to the long-requested optional free-form feedback. Sep 29, 2011 at 17:47
  • Beautiful, so we've actually come to the point of admitting that users are optimizing for a number. Seems like the remainder of that argument is a tautological one, insisting that the number is useful because people want to push it up, and people want to push it up because it's a useful number. Of course neither of those things are supported by any evidence (more flags != better flags).
    – Aarobot
    Sep 29, 2011 at 18:44
  • @Gilles Your other question there sums up my feelings on the evolution of flag weight very well.
    – user154510
    Sep 29, 2011 at 18:46
3

I think that the points about this being a relatively effective automated system (and definitely more effective than any manual system) really make the best point. If there is nothing which points out quickly that a certain user is an ineffective flagger, then the moderator will need to spend that much more time actually analyzing a user's behavior. So, flag weight would still need be a feature for the moderators.

On the other hand, if someone does not receive feedback about making bad/good flags, that person is more likely to continue the destructive behavior. Remember psychology! This is classical conditioning at work! If someone does something good, they get a reward, if they do something bad, they get a little zap.


Now, to propose another solution: use the Slashdot method. Don't tell people their flag weight. Tell them that they are a good flagger, a great flagger, Jon Skeet would be proud, and then, on the other side, "needs improvement" degrading all the way down to "spammer".

9
  • 1
    The point is to get meaningful signal, as opposed to the single bit of signal that currently exists (and maybe not even that, if we're only supposed to use the “decline” button for egregious cases). Sep 18, 2011 at 20:53
  • I added a proposal for an alternate method which gives feedback to both users and mods while providing the same "encouragement/discouragement." Sep 18, 2011 at 20:56
  • But that requires keeping track of valid/invalid flags. I want to get rid of that. Sep 18, 2011 at 20:59
  • 4
    I think that will cause far more problems than it can possibly solve. Sep 18, 2011 at 21:05
  • 1
    SO worked fine before flag weight. “Will cause far more problems than it can possibly solve?” Show me one problem that flag weight solves (apart from the SO queue priority, for which far better solutions have been proposed). Sep 18, 2011 at 21:20
  • 1
    On what basis do you claim that the current system is effective? The premise on which you base your answer is precisely what's under dispute.
    – Aarobot
    Sep 18, 2011 at 22:13
  • @gilles the huge increase in flagging, driven by flag weight and the badges, has led to us directly removing tens of thousands of off-topic and very low quality posts. Stack Overflow did not work "fine" before flag weight -- there was just a huge mass of hidden, putrid content on the site that nobody was helping us curate. Sep 29, 2011 at 9:24
  • 1
    @aaro see Lance's answer -- "I find myself pushing a bit more to flag, going through the review list when I have a free minute, because there is a badge at the end"; the flag weight system is effective because it massively incentivizes flagging. More flags = more removed low quality content = better experience for everyone. The data is incontrovertible on this. Sep 29, 2011 at 9:26
  • @Jeff, that's a red herring. Of course it incentivizes flagging but that doesn't mean that there aren't several other ways to create the same incentives without also creating all the nasty side-effects of a public flag weight. It's not necessary to ding users at all for bad flags; just give out a badge after X actioned flags and issue warnings to egregious abusers.
    – Aarobot
    Sep 29, 2011 at 18:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .