Off-Topic reason usage and statistics

Question

Now that the individual Stack Exchange sites have been given the ability to set three custom Off Topic reasons per site is there a way (or intention to provide a way) to analyze which of these reasons are being used, how often, and which (if any) aren't being used at all?

The moderators have the ability to change these reason codes should the situation arise, and I imagine this situation would be when a more common OT reason is being manually entered in the 'other' field more than some of the preset ones are being chosen, but how are we to know whether some reason codes are being used and others aren't?

It could be that the same - or versions of the same - custom reason code is being manually entered quite frequently (admittedly as this is free-text it's going to be hard to provide any analytics on this, aside from just counting how often the 'other' reason is chosen) but questions get closed (aka "on-holding") by mods, by the community or a combination of both on a daily basis so it could be difficult to track what codes are getting used more than others.

Whether this means we need specific analytics tools, or just some search criteria to use to find questions closed against one or another of the close reasons, I don't know, but something would be useful.

Is there a way for the site mods to monitor what reason codes are being used for closing questions?

Answer 1

This is something that were watching very carefully, not only in the frequency of application but also the accuracy. We want to make sure that the custom reasons are actually applicable beyond edge cases, and that they're being used as intended. Ideally, these are vetted enough that they would not go stale once activated - with great power and all.

It's really a bit too early to say in what way the data would be built into the interface for moderators to analyze. As they have direct control over these reasons, they will obviously need a mechanism for feedback so they know when and how to tweak them. Most useful feedback will come from their respective meta sites, but they should know if something is gathering dust prior to someone asking "Can anyone remember the last time we closed something as [reason]?"

Right now, that would come from us as we continue to monitor it. However it would probably make sense to simply place something like:

Used NN times in the last 90 days

.. next to the close reason in the same interface where they edit or deactivate it. That would of course probably need to be heavily cached, but when you're talking about 90 days, that's not a big deal.

But, I can't yet say for sure what is going to be put into place, because it's our first time looking at the data too. It's just a little too soon to think about it. For now we'll let them know if we see something with extremely low usage. Moving forward, we'll probably get something more convenient for all bolted on.

Answer 2

From what I understand, @Shog9 et. al., are looking at this very closely and will use that to tweak site specific off topic reasons for Stack Overflow (I don't speak for SE, and I'm sure they'll see this question. I will also link to it in the Stack Overflow Moderator Chat Room).

I can tell you that right now, Moderators for Stack Overflow do not have any extra visibility into what custom reasons are being used (And in fact, we've been removed from the close process entirely for normal flags (not custom flags)).

I think in the longer term, we should have this visibility (and I gather from the conversations I've had/seen that this is on their agenda), but it's still very new, so there's no tooling for moderators currently around this.