Flag weight should be calculated in terms of % correct, not as though it were voting [closed]

Question

Stemming from discussion at Flag weight 500: How can you tell if you're still flagging correctly?

Statistically speaking, the current flag weight system is broken.

If I get 51% of my flags correct, ultimately I will reach 500. Shouldn't someone who flags correctly 90% of the time have their flags shown in the list first?

Conversely, if I flag 49% of the time I will reach 0. Shouldn't I be more highly ranked than someone who only flags correctly 10% of the time?

Why is it that someone who flags correctly 51% of the time gets to be at 500, while I'm stuck wallowing in the mud because I'm flagging correctly 49% of the time?

Flag weight also affects how many flag votes you have daily. If you flag well only 51% of the time, should you really be getting all the bonus flags the weight of 500 gives you?

---

Solution

I'd rather have a system that uses a sliding window of, say, the last hundred flags. How many of those were acted upon? It would easily translate into a percentage, and be very easy for the user to understand how well they are currently flagging.

It's technically more difficult to implement a sliding window than the current system, so an alternative system would have much the same effect, and should be easy to implement:

The higher the weight a user achieves, the less effect a good flag has on weight, and the more effect a bad flag has on weight, and vice versa.

So at 500 weight, good flags have no effect anyway, but a bad flag might drop the weight by twenty points or more. It would then take 10 good flags at two points each to get back up to five hundred. At the bottom, a single good flag will jump the user up by twenty points, and it would take ten bad flags to get them back to zero.

In the middle, each good and bad flag would be ten points up or down. In this way, users would tend to stay near the middle unless they had a better or worse than average flagging record, and if their average is under ninety percent, they won't be at 500.

It would be easier to implement than a sliding window, and it would still give a useful effect. One couldn't get to five hundred, then simply maintain a fifty percent average and stay at the top of the list.

A simple implementation:

A good flag: currentweight += ( 500 - currentweight ) / 25

A bad flag: currentweight -= currentweight / 25

Rounded up, so at four ninety nine you'll still get one point for a good flag to get you to five hundred, and vice versa.

---

As a comparison:

If you maintain a 60% flagging average, then it'll take you about 180 flags to get to 500, but you'll get there eventually. You'll also get 20 additional flag votes per day. If moved to the proposed system, you'd have about 275 to 330 flag weight, and it would only take about 30 flags at this rate to get to that point.

This means that not only does the system more accurately portray a user's true flagging accuracy, but it resolves more quickly, thus reacting to user trends more quickly, and giving better feedback to the user as the whether they are flagging well or not.

Accuracy  Current Weight/Bonus votes  Proposed Weight/Bonus votes
10%       0              0            50              0
20%       0              0            100             0
30%       0              0            150             2                   
40%       0              0            200             5      
50%       100            0            250             7        
60%       500            20           300             10         
70%       500            20           350             12         
80%       500            20           400             15         
90%       500            20           450             17         
100%      500            20           500             20          

This, of course, uses the suggested equations above, while it might make more sense to use a slightly different algorithm, it should be obvious that there's no point to having a "weighting" if it doesn't actually represent a user's flagging accuracy.

Answer 1

Some version of the proposed simple implementation has been implemented: https://meta.stackexchange.com/questions/80170/what-is-flag-weight/80178#80178 Flag weight can now grow up to 750, but beyond 500 the increase is sub-linear; in fact, 750 is only an asymptotic value that can't be reached. The penalty for a bad flag is independent of your current flag weight.

Answer 2

I already commented that I disagree entirely with the premise of the system being broken on account of your example scenarios, as they are unrealistic. People don't reach 500 or get denied from 500 by staying at abysmal success rates - they either shape up, or face consequences. This is a human operated system, after all. But let's get off that subject and discuss your feature request here.

---

I think this places too much weight on occasional points of time, and weakens the impact of trends. Trends are what are more important towards judgment, and single incidents rarely have a strong impact.

When your flagging style is of a consistently great yield such that you're well on your way to 500 weight, it's almost a foregone conclusion that you'll be at 500. The only thing holding you back would be the lack of flaggable content, which is a good thing for the site. Making the process take longer in response to doing well seems to be prolonging the journey for no real benefit. If all of your flags are excellent, does it really matter if it takes 1 or 20 to complete the last leg? In the end, your next 20 are all going to be great anyway.

Consequently, the overpenalty on a missed flag is harsh on people. Mechanically, things between 450 and 500 are so close that it's largely unimportant to the system. But for people, the thought that tripping up one day will mean your next 10 flags are considered "less important" than those of your former compatriots... it comes off as slightly unfair. We don't expect perfection of these users - all we want are great flags.

The important judgments from flag weight come from trending, which is not really tied to your success rate. Whether you have a 75% success rate or a 100% success rate, a string of 15 bad flags will be equal amounts of bad news. The current system makes this very intuitive in that you'll receive the same penalty. But in the weighted system, the one who falls from a height will suffer a greater penalty, while the person who has a history of bad flags is not hampered as much.

A user's history of flagging success tells us what should be done when they stray from the path of good flagging. But it doesn't really impact the severity of that straying, nor the necessity of action. A former 75% flagger who has redeemed herself is just as capable of a basic trip up as one of the deputies, and I don't see why it should have any more or less of an impact based on their current flag weight.

---

Your revised post brings back the premise and assumption that I think is unhealthy to consider.

A user with 60% flag success rate is not going to make it to 40 flags, nevermind 180, before something has to change. People do not stay at some kind of fixed flag success rate if they're flagging poorly. If you're not simply making the occasional honest mistake, then you're consistently making bad flags. And that will warrant action long before that kind of user can make any progress and attain any significant number of flags per day.

There are two ways that this will go down. Either the user will improve their success rate by not making bad flags, or they will continue to get warnings and eventually suspensions for ignoring the warnings. In the case of the former, they have every reason to be getting more flags per day, because they won't have that abysmal success rate.

In order to reach a decent flag weight, you need to have a good success rate. Otherwise, things unrelated to your good flags will get in the way very easily.

---

Flag weight is designed to illustrate, in very simple numbers, the relationship between a user's consistent good flags and the likelihood that their future flags will be important.

I think your proposal is not a wise idea because it encourages stagnation of growth for low weight users and it, perhaps ironically, undermines the impact and meaning of flag weight as reliability.

We don't want bad flags to come in, nevermind in any sort of trend. Under your system, there is a period of time in which alternately flagging good and bad flags will result in an increase of weight over time. In fact, the more problematic a user's flags, the more lenient the system is in allowing them to make bad flags, to the point that you can actually make multiple bad flags and overcome them with one good flag.

Which draws to the second point, undermining the meaning of flag weight. Someone with a low rate means that we have a slight expectation for their flags to be not useful. Someone with a high rate means that the majority of their flags will be actionable. We expect a trend from that.

We don't want to receive bad flags, but your proposed system is more tolerant if the bad flag comes from someone with a history of flagging badly - this is not the message that we want to send as moderators. Comparatively on the other side of the scale, suppose someone with a very excellent style that produces streams of high quality flags has a bad day and makes 3 one-time mistakes in flagging. These one-time mistakes have a lot more impact on that user's flag weight than the entire history of high quality flags. That seems to very readily discard the intent of flag weight to mean reliability, because we suddenly don't think that user is nearly as reliable.

This is what I meant by my very first statement in this answer: this system places too much weight on occasional points in time, and weakens the impact of trends. As with the reverse, one single great flag does not redeem someone with a history of bad flagging, especially if they're going to just keep at it.

Answer 3

I disagree that the flag weight needs to be calculated in a more useful manner.

A bad flag isn't necessarily a bad flag, it just means that a moderator disagreed with you and dismissed the flag. Nothing needs to be done which will make people more careful about what they're flagging unless the mods are being overwhelmed with low quality flags.

So it's mathematically possible that someone who sends unwarranted flags a lot has the same weight as you. Does that mean someone isn't going to get to yours fairly promptly anyhow? Probably not.

Answer 4

I say no. We don't want flag weight to just become another way to show that you're better than someone else. I can easily see some users reaching 500 and never flagging again for risk of breaking their perfect score because with your implementation, one wrong flag would take 20 more to get back to 500.

Also, what benefit does it gain to penalize the guy with 500 flag weight with -20 for a bad flag while the guy at 30 flag weight would only lose 1?