Allow moderators to reverse and/or nullify flag decisions

Question

We've been getting a lot of complaints about moderator flag decisions being incorrect or inconsistent lately.

• Flag declined, but question closed for the same reason
• Why does flag marking as helpful/declined not always correlate with moderator action?
• Why was this not-a-real-question declined?
• Mark flags as valid if question is subsequently closed for the exact reason (eg not a real question) as the original flag
• Flag moderation "disputed" but question closed anyway
• Bind flags to revisions to prevent unjustified declination resulting in flag weight loss
• Moderators should see the post as I flagged it, not the edited version
• Flag declined after a day
• Inconsistent decisions on flag validity on clear cases
• Declined flags which are later approved should be removed from profile (10k)
• Is there anything broken with the way flags are being handled?

Some of the complaints about specific posts turn out to be unfounded, but the general problem still exists. Sometimes moderators make mistakes.

In part because I'd like the chance to fix my own mistakes, and in part because I'd like the complaining about flag weight to stop (or at least be reduced), I'd like to propose that moderators be able to reverse flag decisions, or at least nullify them in borderline cases. Here's how I picture this working:

1. A user flags a question for moderator attention as "Not a Real Question."
2. A moderator disagrees and dismisses the flag as unhelpful.
3. The community later closes the question as "Not a Real Question."
4. The user now has the option to dispute the flag decision from their flag history console, and three things can happen:
   • The flag can be reversed by a moderator, which means the user gets back the 10 points of flag weight that they lost plus whatever they would gain from a helpful flag at their current flag weight. This would be used in cases where the moderator originally just made a mistake like hitting the wrong button.
   • The flag can be nullified by a moderator, which means the user only gets back the 10 points of flag weight they lost. This would be used (at the discretion of the moderator reviewing the disputed flag) in borderline cases where the moderator may not agree 100% with the flag, but the user shouldn't be penalized for it.
   • The flag dismissal stands as "unhelpful." In order to prevent every single flag that gets dismissed as unhelpful from being disputed, the user will lose an additional 10 points of flag weight if their dispute is not either reversed or nullified.

If this gets implemented, any future complaints here on Meta about flagging decisions should be met with immediate closure and comments instructing the user to dispute the flag.

I'm not 100% convinced of the utility of having both reversal and nullification, so maybe an implementation only needs to include one or the other. (See my update below.) As always, I'm open to ideas and suggestions for improvement.

TL;DR: Moderators should be able to reverse flag decisions, but users should have to risk something so every single decision doesn't get disputed.

---

Matthew Read wrote in his answer below:

Our friendly neighborhood waffles has determined that approximately every 1 out of 6 declined flags, or ~17%, is for a post that gets closed. That's really high. This auto-nullify method would eliminate the trashing of flag weight for users who were probably doing the right thing to flag these to-be-closed questions.

1 out of 6 is really high. A lot higher than I would have guessed, but I was only thinking of those cases where moderators mistakenly dismiss a flag as invalid and the community (or another moderator) later takes the exact action suggested. In a lot of those 1 out of 6 cases the flag is marked invalid purposely to send a signal to the flagger that the post really didn't need to be flagged at all (like when it should have just been downvoted, edited, or a vote to close would suffice). In those cases I'd really prefer if the flag had never happened, so I'm even more in favor of having both the ability to nullify flags and the ability to reverse them.

If we can't get this as a moderator ability to review and correct mistakes, I agree with Matt and waffles that just automatically nullifying all of the flags that are disputable would alleviate most (if not all) of the problem.

Answer 1

Sorry, I am declining this.

Given the recent abolition of flag weight all complaints on meta have ceased.

The decision of what to do with a flag is subjective; it is entirely possible and justifiable for some mods to decline certain flags while others will not. There is no objective measure out there. The hope is that moderators use the decline feature as a tool to teach the community how to better flag.

Obsessing over every little flag is wasting moderators time. I do not think we should build features that encourage this kind of obsessing.

The cost for a declines are now very small. I see no reason to make any changes here.

Answer 2

I'm one of the previous complainers, although I don't appear to be on Bill's list.

Ultimately, we all understand that flag decisions are subjective, so I don't mind so much that some of my flags are declined.

I'd just like to not lose the flag weight if the community subsequently agrees with me. I don't even care about getting the extra point - just don't penalise me.

We can debate and debate whether flag weight is productive or even relevant, but it is used to prioritise flags in the mod queue and is a measure of how appropriately you are flagging. If the community subsequently agrees with you then I don't think it's unreasonable that the 10 point penalty is removed.

I don't much care either way whether you are awarded the ~1 point increase since it's not so hard to get that back with another flag.

In fact, if all that happens is the penalty is removed, then we don't even need to bother anyone to go back and revalidate the flag - the system can handle it automatically.

It's not even necessarily about ego as @JNK suggests - nobody else sees your flag weight so any ego boost is minimal with nobody to compare against.

The problem as I see it with flags being declined, is it makes me less likely to flag in future and since flagging is an important mechanism for maintaining the quality of the sites, I'd think we want as many people flagging as possible. This shouldn't place too much burden on mods, since we have a mechanism that identifies flags from users who consistently flag appropriately and it's called flag weight.

I can speak from experience that I've ignored several cases that I consider appropriate to flag recently and they stay around on the site because I don't want to take the hit in flag weight. I've heard the argument that there are plenty of other users who will flag, but I don't think that's enough. this argument just further reduces the perceived value of flagging since it gives the impression that any one individual's efforts are irrelevant because there are many more people who would do it.

While that may be true, I think it's unnecessarily discouraging.

If nothing is changes (which is fine - it won't stop me contributing), then I vote we remove flag weight altogether but obviously keep flagging.

Answer 3

In a lot of those 1 out of 6 cases the flag is marked invalid purposely to send a signal to the flagger that the post really didn't need to be flagged at all (like when it should have just been downvoted, edited, or a vote to close would suffice)

Adding another answer just to address this particular sentence.

I don't agree with this mentality at all for flags about closing and here's why: most posts on Stack Overflow will never, ever get the 5 close votes necessary to close.

That does not mean that users should never cast close votes, but that there is a harsh, stark reality of:

• how many views will this question get?
• how many of those views will be users who have the 3k rep necessary to cast a close vote?
• how many of those 3k users will understand the purpose of close votes and how to use them at all?
• how many of those users will be willing to cast a close vote on this question?

If the average no-name question gets, say, 20 views in a day.. the odds are looking mighty slim here. Here's a question I picked randomly from page 100 of the questions page:

https://stackoverflow.com/questions/7368595/find-word-in-nth-row-for-column-1-and-put-new-word-in-nth-row-for-column-2

This has 19 views. Now, our view counter is very strict but I can assure you no more than at most 3x that many human beings viewed this question. Out of those 57 hypothetical users, run the questions I asked above. You'll be lucky to get ONE, yes, ONE user who cares enough to cast a close vote in there.

Therefore, flagging in low-view tags for closure -- provided you feel strongly about the close -- is about the only way a close is ever going to happen in practice.

Now, I can see the issue with

Why didn't you just downvote this instead of flagging it?

(though I might suspect in some cases on answers where a downvote costs -1 that would be why, and that's certainly not healthy..)

and

Why didn't you just edit this instead of flagging it?

But even in those cases, unless I have specific reason to believe the person is flagging in seriously bad faith I would still dismiss them as helpful provided the post had something actionable in it.

Of course, using the moderators as proxy workhorses for stuff you're too lazy to do yourself is bad and grounds for flag dismissal. But of all the flags I deal with (a few hundred on weekends) I see that maybe 1/100 of the time!

Answer 4

Edit: The new flagging dialogue is very good. As long as mods stick to Jeff's strategy (only declining when there is compelling evidence that the flagger wasn't trying to be helpful) this should be a non-issue.

---

I recently proposed that flags declined due to an edit occurring after the flag should be automatically nullified. I think the same could work here. If a post is closed for any reason then all prior flags on the question that were declined should be automatically nullified instead.

Our friendly neighborhood waffles has determined that approximately every 1 out of 6 declined flags, or ~17%, is for a post that gets closed. That's really high. This auto-nullify method would eliminate the trashing of flag weight for users who were probably doing the right thing to flag these to-be-closed questions.

---

In the end, is a flag as "Not A Real Question" really so different from a "Not Constructive"? I don't think so, and the difference is certainly not big or important enough to warrant multiple reviews of the flag. When things get complicated, I don't think we should agonize over the outcome. Just stick with the middle ground (nullifcation, no flag weight change) and let that be done automatically as I propose.

Users can still learn from helpful and declined flags, and shrug at nullified flags. (Which is not to say that the occasional obsessive won't get upset by a nullification, but it will be much less frequent, and more easily dismissed as unjustified whining. And mods can't be blamed since the nullifcation is automatic.)

Answer 5

Wouldn't it be easier just to hide flag weight? It doesn't really matter but has been the cause of an inordinate amount of stress about this.

Just change from showing a numerical weight to a tiered system (maybe expand the existing badge system to add additional levels).

I don't think it's a productive use of moderator time to make sure that users' egos are soothed if they flag a question and it's not acted on. Mods should be dealing with issues, not making sure nobody's feelings get hurt about whether their particular flag was valid or not.

All that SHOULD matter is that appropriate action has been taken. Whether JoeUser gets credit for flagging correctly or not is beside the point, as long as the question/answer is deleted or acted upon correctly.

If you obfuscate the numerical weight behind "grades" like Deputy, Commander, Marshall, Superhero, Will or whatever, users won't obsess over individual flag decisions but will still get gratified that they are progressing. Don't take away the game aspect, just the minute granularity that now exists.

EDIT, based on feedback from Bill the Herpetological Mod:

Alternatively, base the flag weight on final action on the post instead of individual moderator actions...

• A mod rejects the flag but question is closed by the community? Flag weight bump.
• Above question is reopened? Flag weight drop.

Answer 6

I was going to edit my first answer, but the edit ended up being longer then the original so I added a new answer. Bill, et. al. feel free to merge it with my previous answer if you think that would be more appropriate...

Bill hit the nail on the head in an earlier comment with

As a user, those dismissed flags that are later justified by community action must be the most annoying type.

That's exactly the kind that frustrates me, and are really the only declined flags that I object to; after all, if I flag and the question is never closed/migrated, or the answer removed/converted to a comment, then it was almost certainly a bad flag, or borderline enough that it becomes subjective, or maybe I was having a bad day and overreacted. I'll get over it. The mods do a great job and as a community driven site, there are going to be differences of opinion, I can live with that. I guess the real annoyance for me is when you start building up flag weight, the penalty remains the same while the boost for an accepted flag decreases.

I understand the reasons for the boost decreasing, and actually agree with it - we've already got reputation so we don't need another ever-increasing metric as a signal of how much we interact with the community.

I do have two different suggestions for a solution to the frustration, both of which the system could handle without increasing the load on any mod:

• Have the flag weight penalty for a declined flag be equal to a number of accepted flags. For example, when you start out each accepted flag is worth 10 and a declined flag is worth -10. For me right now, a declined flag is still worth -10, but an accepted is worth around 1.1. This means that for me, a declined flag has a much greater relative impact than for someone new to flagging. This doesn't seem right to me - I've already demonstrated I can flag reasonably well since my flag weight is currently 626.4. A new flagger with no proven history of good flagging gets a penalty of one good flag (-10) but I get a penalty of about 9 good flags, despite having shown that I flag reasonably responsibly. Let's say the penalty was 5 flags rather than 10 points - the new flagger would get a penalty of 50 and mine would be about 5.5; seems unfair, right? Except for both of us, that would only require 5 good flags to reverse.
• Or, as a variation, how about having the flag penalty asymptotically approaching a low number (maybe zero, but more likely two, or three so it still smarts a little, but doesn't necessarily feel punitive) as the maximum flag weight asymptotically approaches 750? This would allow the algorithm to factor in the fact that users with higher flag weights have necessarily demonstrated their ability to flag responsibly and this is reflected in their declined flags getting a lower penalty than a user who has no track record.

A possible problem with the first is it may discourage flagging since it could be perceived as unfairly penalising new flaggers. If flag weight was a representative overall good flag value then this wouldn't be an issue since the penalty would no longer be relatively disproportionate at one end of the scale.

Personally, I think I prefer my second option as it can make allowances (admittedly in a simplistic manner) for a user's previous good flagging behaviour, but still includes a penalty for declined flags.

Answer 7

Well, I think the problem here is that too many flags were being declined. Declining a flag should be exceedingly rare.

(and it didn't help that the language we used before we changed to helpful/declined was valid/invalid).

We've improved the flag decline process in two ways:

1. It is more steps to decline than mark helpful, which properly reflects the amount of effort that should go into a decline.

2. There are reasons for a decline, which will be presented to the user automatically if they look at their flagging page. These reasons also help educate mods about proper decline scenarios for flags.

Like so:

Then, if you select "declined..." you must select a reason:

The current default decline reasons are:

• flags should not be used to indicate technical inaccuracies, or an altogether wrong answer
• a moderator reviewed your flag, but found no evidence to support it
• flags should only be used to make moderators aware of content that requires their intervention
• other ...

"Other" allows typing in an arbitrary sentence of ASCII explaining why the decline was necessary.

Answer 8

I don't care about rep or flag weight. I just want to contribute, and not waste my time doing so. My most recent flag was denied, and I've been wasting the past half hour trying to figure out why, looking up flag disputes and landing here, and just trying to submit a better flag.

This question had been flagged as a duplicate since 2016, but hadn't been closed. I flagged it for speed close because I've done that in the past with similar questions, and they did get speed closed.

This time though, I drew the short straw and an unnamed moderator (accountability and transparency anyone?) declined my flag like this:

I find it hard to believe that both myself and the 2016 user who initially flagged this question as a duplicate failed to notice the two questions were for different languages. Anyway, I wanted to see if I was wrong. However, that mod also deleted the comment that marked the question as a dupe.

Worse, when I attempt to flag the question as a dupe now in order to see what duplicates SO suggests, I can't. The option is gone:

Other reasons to dispute flags

So there might be other reasons to dispute flags, such as mods making human mistakes and making it impossible to flag questions as duplicates. Or maybe I made the mistake, but later found a legitimate question that was duplicated by this one.

Is there no recourse?