Share the results of graduation reviews with the community

Question

One of the questions that is inevitably asked on every beta site at some point, often more than once, is a variation of "why haven't we graduated yet?" or "what do we need to do to graduate?".

The whole process around graduation is rather intransparent. I've been a moderator during the whole beta phase until graduation on one site, and I'm a moderator on another beta site that hasn't graduated yet. I've quite a bit more experience with the process than the average SE user, but I still can only guess why certain sites have not graduated yet and why some sites have.

What I know about the process is that in certain unknown intervals the SE community team evaluates a site and makes a decision on whether to graduate it or not. The general requirements are a minimum threshold of activity on the site, the absence of any severe quality issues, a sufficient amount of high-rep users to ensure moderation and a healthy growth trend in traffic.

The feedback from SE on graduation is currently rather minimal, unless someone explicitly asks, there is no feedback at all until graduation. There are exceptions like severe quality issues that jeopardize the existence of the site, but most of the time there is no feedback. Even if someone asks, the answer are often rather vague.

I think that more feedback on graduation would be very useful to the beta sites, and I'd like to propose the following idea on how SE could provide better feedback without too much additional effort.

The first point is that the schedule on when sites are evaluated should be public, or at least available to the moderators. The result of each evaluations should be communicated to the moderators. This doesn't have to be in any polished form, a very simple summary would already be helpful. Even a simple "no major quality problems, nice question volume but not enough high-rep users to moderate, traffic still a bit low" would be a large help. It shouldn't be more additional effort than whatever report is written internally anyway.

I'm proposing to communicate this to the moderators mainly to reduce the additional effort for SE. The internal communication can be less formal, and the moderators can then decide how to inform the community, if it makes sense (if there is no change compared to the last evaluation, that is probably not necessary).

Answer 1

I'm going to answer with where we are in contemplating this. Nothing in this answer is carved in stone. We're still very much up in the air about what we can do here as well as our process for graduating sites as a whole. This needs a response from us, and the best I can provide right now is a brain dump of where we are with it.

We've discussed something similar to this being opened on the child meta site once we've decided to put a site into the graduation queue:

Hello! I'm John Smith, a community manager at Stack Exchange, and I have some important and exciting news to convey. After recently completing a review of [site], we've determined that you're ready to graduate. Preparations are now underway for this to happen, and [site] is now in a queue for the designers to work on.

While we're actively trying to find more talented designers, there's only a few on staff - it might six months or longer for design concepts to be posted here on meta for your feedback, which is the next step in the graduation process. Please be patient!

What I'd like to do now is open a discussion to see if anyone has any questions or concerns regarding the changes that come with graduation, or any observations gained from participating in the [site] beta process that you'd like to share.

This is not the actual text that we would use, this is something off the top of my head that captures the key points that we'd like to convey.

But, there's problems with this ...

It could be a year before you actually graduate

Good designers are very difficult to find. If we found 10 great designers, we'd hire all 10 of them on the spot. They have to work on whatever is strategically important for the company at any given time, most of the time that's going to be getting new graduated sites launched - but we have careers, advertising and internal products that need them too.

Barring the discussion that we'd open at the end, we're basically coming to you and saying "We think you're awesome, and good things will eventually happen for you." - that's not much different than anything else we say throughout the process.

While it's intended to tell you that you've crossed a metaphorical finish line, it's also telling you that you have to wait months or longer to get your ribbon. That's less than ideal, and it turns what's currently a major event into something sort of 'meh'.

They got their message when [stats], why haven't we gotten one yet?

Even before coming to work here, I've urged folks to stop looking at those stats so much - while they do allow us to see when a site has consistently good and climbing levels of participation, they aren't the only thing that we look at. This is not an exhaustive list:

• Are there unresolved concerns around content policy, or something else that could cause the site to shift course?
• Are there any quality issues that concern us?
• Do new users tend to stick around?
• How long does it take for stuff to get answered on average?
• Is a noticeable percentage of the content basically copied (with attribution) from Wikipedia or other places?
• Have we created something better than most other sites surrounding this topic?
• Can they sustain an election?
• Can they sustain a shift in privilege thresholds?

... and possibly 50 more things that come up during the course of evaluating the site, it really depends on the topic. Those conversations happen among us at first, because it's hard enough to conclude such a discussion with 10 involved, opening that up to a few thousand would be ... a mess (it's purely for practical reasons).

New sites do better when they focus on what they're doing well. We may have concerns, and notice that folks on meta share the same concerns and are actively working on them. There'd be little point in us coming to you with what you already know.

If there are any serious concerns that come out of an evaluation and we don't see you talking about them, we do bring them to your meta site, usually the same or next day. But we're not going to frame that as "Y'all scared off all the noobs, so we're punishing you by not letting you graduate" and if we even mentioned that these concerns came out of a graduation review, many would see it like that.

Put simply, the absence of such communication from us after a period of time is very likely to cause unnecessary angst. Some sites have remained in beta for several years.

Our current system isn't optimal, at all.

A few on the team are diving pretty deep into historical data to find correlations of characteristics of sites as they were when we graduated them. This is in no way an attempt to automate the process of graduating sites based on data alone, but more of a surety that we don't miss sites that might be ready, where 'ready' is only apparent with historical context.

Remember me previously mentioning 10 people talking about these? That's also a problem, it's nearly impossible to get 10 people to agree on anything - so there's a sort of majority system. The problem is, these discussions can inject trepidation for some that wasn't there previously - and part of that is good - humans are prone to oversight. But it's not optimal, and until we make it optimal, I think it would be a mistake for us to suck you into what is currently a rather crazy process.

tl;dr;

We'd like to do this, but it's first going to require a pretty large revamp of how we approach the process, because we're going to be bringing you into that process much sooner than later by doing so, and I'm not certain that's advantageous given how it currently works.

We want to arrive at a system where graduation is less opaque, and that's one of the things we're going to be talking about a week from now when everyone is in Denver.

Note, I'm not saying yes, no, maybe - I'm just letting you know where things are internally.