What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

Stemming from discussion at Flag weight 500: How can you tell if you're still flagging correctly?

Statistically speaking, the current flag weight system is broken.

If I get 51% of my flags correct, ultimately I will reach 500. Shouldn't someone who flags correctly 90% of the time have their flags shown in the list first?

Conversely, if I flag 49% of the time I will reach 0. Shouldn't I be more highly ranked than someone who only flags correctly 10% of the time?

Why is it that someone who flags correctly 51% of the time gets to be at 500, while I'm stuck wallowing in the mud because I'm flagging correctly 49% of the time?

Flag weight also affects how many flag votes you have daily. If you flag well only 51% of the time, should you really be getting all the bonus flags the weight of 500 gives you?


Solution

I'd rather have a system that uses a sliding window of, say, the last hundred flags. How many of those were acted upon? It would easily translate into a percentage, and be very easy for the user to understand how well they are currently flagging.

It's technically more difficult to implement a sliding window than the current system, so an alternative system would have much the same effect, and should be easy to implement:

The higher the weight a user achieves, the less effect a good flag has on weight, and the more effect a bad flag has on weight, and vice versa.

So at 500 weight, good flags have no effect anyway, but a bad flag might drop the weight by twenty points or more. It would then take 10 good flags at two points each to get back up to five hundred. At the bottom, a single good flag will jump the user up by twenty points, and it would take ten bad flags to get them back to zero.

In the middle, each good and bad flag would be ten points up or down. In this way, users would tend to stay near the middle unless they had a better or worse than average flagging record, and if their average is under ninety percent, they won't be at 500.

It would be easier to implement than a sliding window, and it would still give a useful effect. One couldn't get to five hundred, then simply maintain a fifty percent average and stay at the top of the list.

A simple implementation:

A good flag: currentweight += ( 500 - currentweight ) / 25

A bad flag: currentweight -= currentweight / 25

Rounded up, so at four ninety nine you'll still get one point for a good flag to get you to five hundred, and vice versa.


As a comparison:

If you maintain a 60% flagging average, then it'll take you about 180 flags to get to 500, but you'll get there eventually. You'll also get 20 additional flag votes per day. If moved to the proposed system, you'd have about 275 to 330 flag weight, and it would only take about 30 flags at this rate to get to that point.

This means that not only does the system more accurately portray a user's true flagging accuracy, but it resolves more quickly, thus reacting to user trends more quickly, and giving better feedback to the user as the whether they are flagging well or not.

Accuracy  Current Weight/Bonus votes  Proposed Weight/Bonus votes
10%       0              0            50              0
20%       0              0            100             0
30%       0              0            150             2                   
40%       0              0            200             5      
50%       100            0            250             7        
60%       500            20           300             10         
70%       500            20           350             12         
80%       500            20           400             15         
90%       500            20           450             17         
100%      500            20           500             20          

This, of course, uses the suggested equations above, while it might make more sense to use a slightly different algorithm, it should be obvious that there's no point to having a "weighting" if it doesn't actually represent a user's flagging accuracy.

share|improve this question
    
@Pollyanna: I like your idea "for high flag weight, bad flags will cause a heavier drop". I'd actually make this gradual; just inventing numbers: 15 for >300, 20 for >400, 25 for =500. In a comment to your recent answer to Martin Clayton's question, Michael Mrozek asked "Why is someone punished more for a mistaken flag if they have a history of good flags?" My answer would be: because high flag weight means high responsibility; mods might start to trust you blindly. –  Hendrik Vogt Mar 7 '11 at 17:34
3  
@Hendrik Speaking as a mod, I honestly don't pay that much attention to who makes a flag unless it's a bad flag, or if the flagger is important towards addressing the job. The former I pay attention to so I can notice if there are trends. If they're doing a good job, they'll just be making their way to 500, but it doesn't particularly mean that their flags are always valid. –  Grace Note Mar 7 '11 at 17:37
    
AFAIK all flag weight does is assist in sorting items in the moderator queue. In retrospect I wonder if it should be displayed at all, as displaying it gives the appearance that it is more important than it actually is. –  Robert Harvey Mar 7 '11 at 17:46
2  
@Robert I argued that initially, but flag weight now also affects how many flags per day you get. Even if it weren't publicly displayed, it's easily derived so it's mostly pointless to hide it at this point. –  Grace Note Mar 7 '11 at 17:52
    
@Grace: OK, since your speaking as a mod, it seems my arguments was a weak one. Thanks for the feedback! –  Hendrik Vogt Mar 7 '11 at 18:45
    
My main issue with the weighting as it is (as a flagger) is that it serves as a signal, then abruptly stops. It would be ideal, if we're going to have such a signal, if it increased the amount of valid flagging, reduced the amount of bad flagging, saved mods time on mailing consistently bad flaggers, and reassured those who flag well that they are not wasting moderators time. I don't think feedback on every flag raised would make sense - just a guide is enough. –  martin clayton Mar 7 '11 at 19:13

4 Answers 4

up vote 2 down vote accepted

Some version of the proposed simple implementation has been implemented: http://meta.stackexchange.com/questions/80170/what-is-flag-weight/80178#80178 Flag weight can now grow up to 750, but beyond 500 the increase is sub-linear; in fact, 750 is only an asymptotic value that can't be reached. The penalty for a bad flag is independent of your current flag weight.

share|improve this answer

I already commented that I disagree entirely with the premise of the system being broken on account of your example scenarios, as they are unrealistic. People don't reach 500 or get denied from 500 by staying at abysmal success rates - they either shape up, or face consequences. This is a human operated system, after all. But let's get off that subject and discuss your feature request here.


I think this places too much weight on occasional points of time, and weakens the impact of trends. Trends are what are more important towards judgment, and single incidents rarely have a strong impact.

When your flagging style is of a consistently great yield such that you're well on your way to 500 weight, it's almost a foregone conclusion that you'll be at 500. The only thing holding you back would be the lack of flaggable content, which is a good thing for the site. Making the process take longer in response to doing well seems to be prolonging the journey for no real benefit. If all of your flags are excellent, does it really matter if it takes 1 or 20 to complete the last leg? In the end, your next 20 are all going to be great anyway.

Consequently, the overpenalty on a missed flag is harsh on people. Mechanically, things between 450 and 500 are so close that it's largely unimportant to the system. But for people, the thought that tripping up one day will mean your next 10 flags are considered "less important" than those of your former compatriots... it comes off as slightly unfair. We don't expect perfection of these users - all we want are great flags.

The important judgments from flag weight come from trending, which is not really tied to your success rate. Whether you have a 75% success rate or a 100% success rate, a string of 15 bad flags will be equal amounts of bad news. The current system makes this very intuitive in that you'll receive the same penalty. But in the weighted system, the one who falls from a height will suffer a greater penalty, while the person who has a history of bad flags is not hampered as much.

A user's history of flagging success tells us what should be done when they stray from the path of good flagging. But it doesn't really impact the severity of that straying, nor the necessity of action. A former 75% flagger who has redeemed herself is just as capable of a basic trip up as one of the deputies, and I don't see why it should have any more or less of an impact based on their current flag weight.


Your revised post brings back the premise and assumption that I think is unhealthy to consider.

A user with 60% flag success rate is not going to make it to 40 flags, nevermind 180, before something has to change. People do not stay at some kind of fixed flag success rate if they're flagging poorly. If you're not simply making the occasional honest mistake, then you're consistently making bad flags. And that will warrant action long before that kind of user can make any progress and attain any significant number of flags per day.

There are two ways that this will go down. Either the user will improve their success rate by not making bad flags, or they will continue to get warnings and eventually suspensions for ignoring the warnings. In the case of the former, they have every reason to be getting more flags per day, because they won't have that abysmal success rate.

In order to reach a decent flag weight, you need to have a good success rate. Otherwise, things unrelated to your good flags will get in the way very easily.


Flag weight is designed to illustrate, in very simple numbers, the relationship between a user's consistent good flags and the likelihood that their future flags will be important.

I think your proposal is not a wise idea because it encourages stagnation of growth for low weight users and it, perhaps ironically, undermines the impact and meaning of flag weight as reliability.

We don't want bad flags to come in, nevermind in any sort of trend. Under your system, there is a period of time in which alternately flagging good and bad flags will result in an increase of weight over time. In fact, the more problematic a user's flags, the more lenient the system is in allowing them to make bad flags, to the point that you can actually make multiple bad flags and overcome them with one good flag.

Which draws to the second point, undermining the meaning of flag weight. Someone with a low rate means that we have a slight expectation for their flags to be not useful. Someone with a high rate means that the majority of their flags will be actionable. We expect a trend from that.

We don't want to receive bad flags, but your proposed system is more tolerant if the bad flag comes from someone with a history of flagging badly - this is not the message that we want to send as moderators. Comparatively on the other side of the scale, suppose someone with a very excellent style that produces streams of high quality flags has a bad day and makes 3 one-time mistakes in flagging. These one-time mistakes have a lot more impact on that user's flag weight than the entire history of high quality flags. That seems to very readily discard the intent of flag weight to mean reliability, because we suddenly don't think that user is nearly as reliable.

This is what I meant by my very first statement in this answer: this system places too much weight on occasional points in time, and weakens the impact of trends. As with the reverse, one single great flag does not redeem someone with a history of bad flagging, especially if they're going to just keep at it.

share|improve this answer
    
@Pollyanna I'm not actually intending to emphasize reaching and maintaining 500. To me, it's largely unimportant in comparison to just posting good flags. And... you deleted your comment, so I don't know whether you are retracting it or revising it. Ah well, I'll find out. ♪ –  Grace Note Mar 7 '11 at 17:59
    
I still don't see how this newer system would damage stackoverflow. I suspect I'm going to have to clear my head and come back to your posts later to see if I can understand them better. At the moment the only thing I understand you to be saying against the revision is that a different process prevents user extremes from happening, and that as such having this weighting correspond to accuracy on a linear scale is unimportant. But I suspect there's more that you're saying that I'm not absorbing, so I'll revisit it later. –  Adam Davis Mar 7 '11 at 18:20
    
@Pollyanna Hopefully my new revision makes this a bit clearer. –  Grace Note Mar 7 '11 at 18:24
    
@Grace No, it only emphasizes "extreme situations won't happen" What I'm looking for is more along the lines of, "If this system is implemented, it will hurt the moderators/stackoverflow/questions/answers/voting/flagging in these specific ways X1, Y1, and Z1, and will encourage bad user behaviors X2, Y2, Z2, and discourage good user behaviors X3, Y3, and Z3" which, I suppose, could be considered pedantic. I don't disagree with you - perhaps the existing system takes care of extreme cases just fine. What's wrong with getting closer to "flag weight is a measure of how reliably you flag content." –  Adam Davis Mar 7 '11 at 18:31
1  
@Pollyanna I am thoroughly confused as to how you keep seeing this as emphasizing extremes, when I don't really care about extremes and the revision is emphasizing everything in the middle, not in the extremes. It's all done as a measure of current progress, and the system operates very fine towards that. Flag weight does measure how reliably you flag content. Every example you've given me to otherwise feels unrealistic, or doesn't really take into account how the user reached that place. I will supply another revision, then. –  Grace Note Mar 7 '11 at 18:34
    
@Grace Ah, I am framing the conversation incorrectly, sorry. Ignore the first line of my last comment, it's not really conveying what I meant to say. The remainder of the comment is still in play though - it would help me understand why my proposal is poor if you framed your response in terms of, "If this were implemented the following bad things would likely happen..." –  Adam Davis Mar 7 '11 at 18:41
    
@Pollyanna I've tried now, in the final answer break. Hopefully this is clearer this time, haha~. –  Grace Note Mar 7 '11 at 19:16

I disagree that the flag weight needs to be calculated in a more useful manner.

A bad flag isn't necessarily a bad flag, it just means that a moderator disagreed with you and dismissed the flag. Nothing needs to be done which will make people more careful about what they're flagging unless the mods are being overwhelmed with low quality flags.

So it's mathematically possible that someone who sends unwarranted flags a lot has the same weight as you. Does that mean someone isn't going to get to yours fairly promptly anyhow? Probably not.

share|improve this answer
    
But it does mean that if 90% of my flags are acted on, and 60% of their flags are acted on, we both get the same number of additional flag votes each day. –  Adam Davis Mar 7 '11 at 17:42

I say no. We don't want flag weight to just become another way to show that you're better than someone else. I can easily see some users reaching 500 and never flagging again for risk of breaking their perfect score because with your implementation, one wrong flag would take 20 more to get back to 500.

Also, what benefit does it gain to penalize the guy with 500 flag weight with -20 for a bad flag while the guy at 30 flag weight would only lose 1?

share|improve this answer
    
Fwiw: only yourself and the mods can see the weight. –  martin clayton Mar 7 '11 at 19:57
    
@Martin: I know that. That doesn't stop people from bragging about it, or it might just make them feel better to know that they're at 500. You can also tell if you have a higher flag weight than someone if you have the "Deputy" badge and they don't. –  John Mar 8 '11 at 2:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .