Can (or should) the mods be evaluating these decisions more closely than they are?
No. More scrutiny is always better, however, in this case, it's not necessary (nor is it possible), given the site volume and community.
In particular, there seems to be an impression that mods should be doing more work that I personally feel belongs more properly to the community, such as cleaning up spelling errors in posts, or fixing them instead of closing.
I agree completely, the community ought to handle this.
My own experiences
I started off as a mod on Chem. A new beta site. At the time, I used to read almost every post on the main page, and edit/comment when necessary. This was doable, we had around 2 q/day at the time. Nowadays with ~5 q/day, I don't always get to do this, but I try. And my community does some of this for me.
In December I was elected a mod on Physics, a graduated site that was ~2 years old. I quickly learned that moderation here was very, very different. For one, I obviously couldn't sift through every post on a 50 q/day site. My focus here was different, I would patrol the review queues and the flag queues, and then pick off questions from the main page based on their titles. Reading every single post that comes in? I left that task to the community as a whole.
On Physics, with the lack of an actively moderating 10k community, moderators are not "human exception handlers", they handle a lot of the close/10k flags as well. Which is OK by me, since we aren't getting buried in these. Still, I try to leave tailored comments for posts I close (and edit stuff where necessary). I bet that mods are unable to consistently do that on larger sites either, and that's perfectly understandable.
Of course, it would be better if I could vet every post that comes in. But that's not feasible, I have stuff to do in real life as well.
Stack Overflow is that point which is all alone up top. (MSO is the point with 20 mods; it doesn't really count for this analysis).
Let's see if we can get it some neighbors by using a log scale:
Nope. SO is still alone. Poor thing.
An interesting thing to note here is that questions/day exponentially grows as the number of mods increases.
If there are drastic changes in the style of moderation and level of scrutiny while going from ~1.5 questions/day/mod to 10 questions per day/mod (Chemistry to Physics), there must be a huge difference between Mathematics (40 q/day/mod; one of the highest sites for this statistic) and SO (450 q/day/mod).
Ideally, as the size of a site increases, the state of "moderators are human exception handlers" should become closer to reality. On smaller sites, mods handle stuff that the community should do in an ideal situation, including closing and deleting flagged/queued posts. On such sites, the state is more of "Moderators handle what the community can't".
As the community gets more and more involved in moderation, mods themselves can retreat and handle the "exceptions" (mod flags, sockpuppets, etc).
In the case of Close/NAA/VLQ flags, the mods ideally shouldn't be spending time at all. SO is at the stage where, given the size of the community, the majority of CV flags/10k flags will be handled by the community. With the introduction of the close vote queue (and the upcoming 10k queue), this is closer to becoming a reality.
Is there really a problem here? If so, how can it be solved?
Well, sort of. If the net scrutiny on a typical flag is less, and flags are getting misinterpreted or not handled properly, there is a problem1.
But I don't think the problem can be fixed by simply increasing the amount of time moderators spend on flags. The mods already are quite overloaded, and increasing that load may not be the best idea.
Instead, I see two solutions:
- Increase the number of moderators: If we have a couple more elections, the mods will have a much lesser load and can spend more time per flag
- Get the community to participate more in flag handling: As far as I can tell, a lot of the SO higher reps don't even know the rules (I have seen link-only/comment-as-answer from 10k+s), and aren't really interested in moderation. I think the review queues have been an awesome step to fix this (and having a 10k review queue will augment this). Further efforts to educate the community and increase community participation in the moderation of the site will go a big step in fixing this problem.
- Experiment by simply not handling close/10k flags: We're actually currently trying this on Physics, and the results look promising. Instead of immediately handling close vote flags, we're waiting for the community to handle it. If it takes too long, then we clear it up. Posts take a while longer to be closed, but they get closed in the end without much fuss. SO can run the same experiment, including the 10k NAA/VLQ flags in the mix. The reason that the community isn't handling stuff could simply be that the mods are getting there first.
1. I'm not entirely sure of this, and I have no data to back it up. However, I do get a significant amount of used-the-wrong-flag-reason cases on Physics, and I assume this happens on SO, too. At first glance (keeping in mind the flag reason), the post may not look so bad, (for example, a broad question flagged as OT doesn't look smelly when you check for off-topicness) and in the case of SO, I'm pretty sure that mods don;t have the time to check the post for all the types of smelliness (not just by the flagged reason), as that can take up to 5 minutes.