Are Stack Overflow mods spending enough time evaluating each flag?

Question

This question arises from a few off-hand comments I have been seeing lately that there is an impression about the Stack Overflow mods that they are so buried in a sea of flags that they don't take the time to properly evaluate each question or answer that gets flagged.

In particular, there seems to be an impression that mods should be doing more work that I personally feel belongs more properly to the community, such as cleaning up spelling errors in posts, or fixing them instead of closing.

I also see signs that the community expects, for example, mods to evaluate a post in its entirety before deleting an answer that has been flagged "Not an Answer," or to provide a pinpoint ruling on the correct use of that flag, rather than focus on whether the answer actually provides any benefit.

Is my perception correct? Can (or should) the mods be evaluating these decisions more closely than they are? Could the thought process be improved? If so, how?

---

For some references to what I am talking about, see here:

1. "If you are positive that moderator will agree with the flag without even reading the question, shoot it without fear. Anything else, use at your own risk"

2. I understand that SO mods are busy and can't scrutinize flags, but afaict across the rest of the network NAA is exactly what you use in these situations

3. You and I and a hundred others spending far too much time on meta understand that "wrong" includes "totally besides the point". In my limited experience, non-meta folk consider that "Which way to the station?" "Tuesday" doesn't attempt to answer the question. Spell it out.

4. So having a broken image is a valid reason for deletion? One would think you could at least search web.archive.org and re-upload it to stack's imgur

5. This particular example is a bad one, as I happen to agree with the deletion; my concern is in with the practice moderation even being remotely seen as autonomous, much complacency be reinfored with a catch-all canned response that is sounding too old already. Lastly, an addition just after reading your answer, you don't seem to do the process justice: if one doesn't have the time "to read each and every single post all the way through" then don't even touch those posts that can't be decidedly acted upon. Just acting anyway is not an excuse.

Answer 1

In my experience, moderators are by and large doing their jobs just fine. In my view moderators are there to step in when the community is unable to handle a specific situation. If the community can (and does) deal with it just fine on their own, then let the community handle it.

In particular, there seems to be an impression that mods should be doing more work that I personally feel belongs more properly to the community, such as cleaning up spelling errors in posts, or fixing them instead of closing.

That is not a job for moderators as such. Sure, moderators are regular users as well, and if they feel they need to edit as well by all means go ahead. But I would not count this as the duty of a moderator.

I also see signs that the community expects, for example, mods to evaluate a post in its entirety before deleting an answer that has been flagged "Not an Answer," or to provide a pinpoint ruling on the correct use of that flag, rather than focus on whether the answer actually provides any benefit.

If an answer is a bad answer, the community should handle it. We have enough tools at our disposal to do so. Votes are one thing. Comments asking for improvements, or explaining the shortcomings of an answer are another. And there is always the option to edit.

The things I end up flagging as "not an answer" are those posts for which you don't even have to read the question itself to determine it can't be the answer. (If the answer is actually a question for example. Or a comment.)

I do expect moderators to carefully read the flagged post. But if it's necessary to read the question as well, I feel you're entering territory where the community can handle the situation by itself.

Is my perception correct? Can (or should) the mods be evaluating these decisions more closely than they are? Could the thought process be improved? If so, how?

I sometimes have the idea that it's desire of users for Stack Overflow to be a perfect resource which is at fault. That is, if an answer is incorrect, it should be deleted. Because wrong answers harm the site and we only want correct content. So please moderator, delete this obviously wrong answer because it is harmful.

There is nothing inherently wrong with that, but it's simply not practical. Downvotes will suffice. Careful and constructive comments can help as well. But the best defence is a correct answer.

So by and large I don't have the idea that moderators should do more. I do instead often feel that we as a community don't nearly enough use the tools at our disposal.

Answer 2

In particular, there seems to be an impression that mods should be doing more work that I personally feel belongs more properly to the community

I think this is going to be at the core of the question here. Stack Overflow is a site with ~17 million unique visitors per month that has a team of 16 elected moderators (along with a handful of active SE employees). That's a relatively tiny number of people to watch over a site with this much traffic, but it works because of the limited role of moderators on the SE network.

These sites function due to the community determining the quality of incoming content, and because regular members take on voting, closing, and editing tasks. All of these actions have counterbalances, like a required number of votes to close questions, the ability to reopen closed questions, etc. No one person or small group can hijack the site (for long, at least). It's also a lot harder for people to rage at "Nazi moderators ruining this site" when questions are closed through the votes of five normal members.

Moderators are here to handle the things that the community cannot. We have the ability to unilaterally close questions, delete posts so that the community cannot undelete them, and suspend and delete user accounts. These are tasks that should be performed rarely.

When moderators make comments saying that the volume of flags means that we can't spend a lot of time on each, a common response is that we simply need more moderators. Significantly increasing the average time per flag and adding many moderators to balance that isn't a great solution. Ratcheting up the amount of work required to handle each flag would make the position even more burdensome, and would cause burnout among those doing the job. The community very thoroughly vets moderators before they're elected, but there's a chance with every person you add that they may not live up to the position. It's a lot more difficult to override a moderator's errors than those of normal members.

Instead, what I think we mean is that we want people to put in as much effort when flagging as they're asking moderators to invest when acting on them. We want people to help us out, first by only flagging the things that really can only be dealt with by moderators, and then by giving us a clear enough signal as to what needs to be done. This is at the center of the debate about when to flag things as "not an answer": many people are taking the bare minimum amount of time when flagging answers, both in identifying something that absolutely needs to be deleted, and in conveying why it needs to be removed.

For reasons like this, I'm glad that close vote flags will soon be hidden from moderators and will instead feed into the close vote review queue. That's a task that the community can handle well, and will remove a large number of difficult-to-judge flags from our queue.

However, despite the occasional comment about numbers of flags, the flag queue has continued to be much lower on SO than it was before the most recent election, and that level was lower than it was two elections ago. It feels like things are pretty well under control now, but we're always looking for ways to get the community to help us out a little more by making flags easier to act on.

Answer 3

I would generally expect moderator efforts on my flags to reflect my own effort.

• If I see a post that takes me few seconds to evaluate (spam, question posted as an answer, stuff like that) then I expect moderator to make a similarly quick judgement.

• If it takes me half an hour to study the post and carefully spell out a custom message explaining the issues with it, then I would expect a moderator to go beyond a quick glance when handling my flag, especially if they decide to decline it. That is, unless they quickly find a serious mistake in my reasoning in the flag message (happened at least once to me) - in that case, quick decline looks fair enough.

---

A very important thing to keep in mind here is that some flags can be also dealt with by 10K users:

You can... assist our elected community moderators, as you have time, in... flag handling...

Can I see all flags?

No. If a user picks the flag → it needs ♦ moderator attention → other option, the flag... will only be visible to community moderators and Stack Exchange employees...

For the "audience" of the flags that go through 10K review (any flag but Other), I would never expect anything but a few seconds to handle my flag.

I would also never expect them to do an educated, thorough, accurate judgement (they possibly can but I do not expect this to be guaranteed - heck, as a 10Ker at Programmers, I know that certainly applies to myself).

There are many more 10Kers than moderators (few thousands at Stack Overflow), I did not elect them, I know nothing about their moderation skills, I know nothing about their experience in handling flags.

The last but not the least, with flags handled by a diamond moderator, I have an option to challenge their judgement at meta, have it corrected if my challenge is convincing enough and expect mistake to not repeat in the future. With mistakes in flags handled by 10Kers, there's nothing like that.

Answer 4

TL;DR

Can (or should) the mods be evaluating these decisions more closely than they are?

No. More scrutiny is always better, however, in this case, it's not necessary (nor is it possible), given the site volume and community.

In particular, there seems to be an impression that mods should be doing more work that I personally feel belongs more properly to the community, such as cleaning up spelling errors in posts, or fixing them instead of closing.

I agree completely, the community ought to handle this.

My own experiences

I started off as a mod on Chem. A new beta site. At the time, I used to read almost every post on the main page, and edit/comment when necessary. This was doable, we had around 2 q/day at the time. Nowadays with ~5 q/day, I don't always get to do this, but I try. And my community does some of this for me.

In December I was elected a mod on Physics, a graduated site that was ~2 years old. I quickly learned that moderation here was very, very different. For one, I obviously couldn't sift through every post on a 50 q/day site. My focus here was different, I would patrol the review queues and the flag queues, and then pick off questions from the main page based on their titles. Reading every single post that comes in? I left that task to the community as a whole.

On Physics, with the lack of an actively moderating 10k community, moderators are not "human exception handlers", they handle a lot of the close/10k flags as well. Which is OK by me, since we aren't getting buried in these. Still, I try to leave tailored comments for posts I close (and edit stuff where necessary). I bet that mods are unable to consistently do that on larger sites either, and that's perfectly understandable.

Of course, it would be better if I could vet every post that comes in. But that's not feasible, I have stuff to do in real life as well.

Graphs!!

Stack Overflow is that point which is all alone up top. (MSO is the point with 20 mods; it doesn't really count for this analysis).

Let's see if we can get it some neighbors by using a log scale:

Nope. SO is still alone. Poor thing.

An interesting thing to note here is that questions/day exponentially grows as the number of mods increases.

Analysis

If there are drastic changes in the style of moderation and level of scrutiny while going from ~1.5 questions/day/mod to 10 questions per day/mod (Chemistry to Physics), there must be a huge difference between Mathematics (40 q/day/mod; one of the highest sites for this statistic) and SO (450 q/day/mod).

Ideally, as the size of a site increases, the state of "moderators are human exception handlers" should become closer to reality. On smaller sites, mods handle stuff that the community should do in an ideal situation, including closing and deleting flagged/queued posts. On such sites, the state is more of "Moderators handle what the community can't".

As the community gets more and more involved in moderation, mods themselves can retreat and handle the "exceptions" (mod flags, sockpuppets, etc).

In the case of Close/NAA/VLQ flags, the mods ideally shouldn't be spending time at all. SO is at the stage where, given the size of the community, the majority of CV flags/10k flags will be handled by the community. With the introduction of the close vote queue (and the upcoming 10k queue), this is closer to becoming a reality.

Is there really a problem here? If so, how can it be solved?

Well, sort of. If the net scrutiny on a typical flag is less, and flags are getting misinterpreted or not handled properly, there is a problem¹.

But I don't think the problem can be fixed by simply increasing the amount of time moderators spend on flags. The mods already are quite overloaded, and increasing that load may not be the best idea.

Instead, I see two solutions:

• Increase the number of moderators: If we have a couple more elections, the mods will have a much lesser load and can spend more time per flag
• Get the community to participate more in flag handling: As far as I can tell, a lot of the SO higher reps don't even know the rules (I have seen link-only/comment-as-answer from 10k+s), and aren't really interested in moderation. I think the review queues have been an awesome step to fix this (and having a 10k review queue will augment this). Further efforts to educate the community and increase community participation in the moderation of the site will go a big step in fixing this problem.
• Experiment by simply not handling close/10k flags: We're actually currently trying this on Physics, and the results look promising. Instead of immediately handling close vote flags, we're waiting for the community to handle it. If it takes too long, then we clear it up. Posts take a while longer to be closed, but they get closed in the end without much fuss. SO can run the same experiment, including the 10k NAA/VLQ flags in the mix. The reason that the community isn't handling stuff could simply be that the mods are getting there first.

^{1. I'm not entirely sure of this, and I have no data to back it up. However, I do get a significant amount of used-the-wrong-flag-reason cases on Physics, and I assume this happens on SO, too. At first glance (keeping in mind the flag reason), the post may not look so bad, (for example, a broad question flagged as OT doesn't look smelly when you check for off-topicness) and in the case of SO, I'm pretty sure that mods don;t have the time to check the post for all the types of smelliness (not just by the flagged reason), as that can take up to 5 minutes.}

Answer 5

I guess it somewhat relates to our tools. If you go to the mod tools, the three most prominent metrics are total flags, flags handled and average time to handle. These encourage quantity and speed. And our tools are built to support this. They are not built to support comprehensive feedback to our users or viewing a flag in greater context. Both things that would likely improve quality of flag handling.

What's worse though is that our main effort is a flag queue full of common issues and not those "exceptional conditions that […] can bring your entire community to a screaming halt" as outlined in our theory of moderation. Right now, we are not the human exception handlers we claim to be. We are more like assembly line workers. And I fully agree that most of our current flag handling belongs to the community instead.

VLQ is a perfect example for flags that should not get to us. If you can edit quality into the post, then do that. Don't defer that to me. If the post cannot be salvaged by an edit, then it's pointless to flag it because we cannot magically make it quality. Downvote and/or delete vote it. That's what we will do as well.

Likewise, why do mods have to handle NAA? I trust our users to be able to spot NAA when they see it. Same routine then: downvote and/or delete vote. It doesn't need a flag. Yes, I know being able to delete vote takes a fair amount of reputation. So maybe that needs to be changed.

But as long as we allow people to defer what we expect them to do to us, things won't change. Redefining what can be flagged to us and rewarding community moderation (either through rep or a separate community trust rating) would leave more time for mods to handle the stuff the community at large cannot reasonably handle, e.g. talking to problematic users, stopping spammers, merging posts and finding cheaters, etc. - note that this doesn't mean that mods cannot or should not handle flags at all. They just shouldn't have to do them as part of their moderator role but rather as part of their regular community user role.

TL;DR Concluding from this, I'd say it's not so much a question whether moderators should be spending more time on the existing flags, but rather a question on what moderators should be spending time on at all. It's the what. Not the how.