33

One of the questions that is inevitably asked on every beta site at some point, often more than once, is a variation of "why haven't we graduated yet?" or "what do we need to do to graduate?".

The whole process around graduation is rather intransparent. I've been a moderator during the whole beta phase until graduation on one site, and I'm a moderator on another beta site that hasn't graduated yet. I've quite a bit more experience with the process than the average SE user, but I still can only guess why certain sites have not graduated yet and why some sites have.

What I know about the process is that in certain unknown intervals the SE community team evaluates a site and makes a decision on whether to graduate it or not. The general requirements are a minimum threshold of activity on the site, the absence of any severe quality issues, a sufficient amount of high-rep users to ensure moderation and a healthy growth trend in traffic.

The feedback from SE on graduation is currently rather minimal, unless someone explicitly asks, there is no feedback at all until graduation. There are exceptions like severe quality issues that jeopardize the existence of the site, but most of the time there is no feedback. Even if someone asks, the answer are often rather vague.

I think that more feedback on graduation would be very useful to the beta sites, and I'd like to propose the following idea on how SE could provide better feedback without too much additional effort.

The first point is that the schedule on when sites are evaluated should be public, or at least available to the moderators. The result of each evaluations should be communicated to the moderators. This doesn't have to be in any polished form, a very simple summary would already be helpful. Even a simple "no major quality problems, nice question volume but not enough high-rep users to moderate, traffic still a bit low" would be a large help. It shouldn't be more additional effort than whatever report is written internally anyway.

I'm proposing to communicate this to the moderators mainly to reduce the additional effort for SE. The internal communication can be less formal, and the moderators can then decide how to inform the community, if it makes sense (if there is no change compared to the last evaluation, that is probably not necessary).

  • 6
    Also don't forget the 1-year period between getting eligible for graduation and actual graduation, where at least some information or update on the graduation queue might be helpful, too, be that only on a monthly or even quarterly basis. – Chris says Reinstate Monica Oct 9 '14 at 9:51
  • 2
  • 7
    Yes, please. We've been told for almost a year now that Computer Science was basically ready to graduate, but then nothing (but repetition of the same, and the sad explanation that visual layout is holding graduations back). – Raphael Oct 9 '14 at 16:10
  • 1
    & how about a list of what changes due to graduation? it is possibly overrated. it presumably has little immediate effect on site traffic. what exactly are the benefits? on "con" side it does have major effect/ impact of decreasing privileges for many because of higher rep threshholds. – vzn Oct 9 '14 at 18:35
  • @vzn It is not just a technical matter of immediate system changes in the framework, it is also a rather "prestigious" (for lack of a better term) question of being a full-fledged first class citizen vs some possible site under evaluation. – Chris says Reinstate Monica Oct 10 '14 at 14:08
  • CR understand that already, but that is vague! underlining/ reinforcing my point! – vzn Oct 10 '14 at 14:57
  • 2
    This is not a duplicate, my proposal is that SE communicates the specific evaluations they already perform for each site. The duplicate target is about general criteria for graduation. – Mad Scientist Oct 10 '14 at 15:03
  • 1
    Yeah, the problem Aarthi and I ran into back in the day when we tried to make a meta post after every evaluation is that 99% of the time there's nothing new to say, so we felt very much like a broken record especially on sites where these evaluation posts (back when they were more frequent) were a solid chunk of the meta activity. Communicating just to moderators seems easier and would also allow people who are most familiar with the community to spur any action (if needed). I like this in principle, but since I'm no longer a CM, I defer to them for an actual response. – Adam Lear Oct 14 '14 at 23:45
  • Also, automatically posting the results of each self-eval to meta was meant to address some of this. Whether it does or not is debatable, and points to the fundamental issue with these evaluations -- we need to find a way to do them better, clearer, etc. – Adam Lear Oct 14 '14 at 23:47
9
+50

I'm going to answer with where we are in contemplating this. Nothing in this answer is carved in stone. We're still very much up in the air about what we can do here as well as our process for graduating sites as a whole. This needs a response from us, and the best I can provide right now is a brain dump of where we are with it.

We've discussed something similar to this being opened on the child meta site once we've decided to put a site into the graduation queue:

Hello! I'm John Smith, a community manager at Stack Exchange, and I have some important and exciting news to convey. After recently completing a review of [site], we've determined that you're ready to graduate. Preparations are now underway for this to happen, and [site] is now in a queue for the designers to work on.

While we're actively trying to find more talented designers, there's only a few on staff - it might six months or longer for design concepts to be posted here on meta for your feedback, which is the next step in the graduation process. Please be patient!

What I'd like to do now is open a discussion to see if anyone has any questions or concerns regarding the changes that come with graduation, or any observations gained from participating in the [site] beta process that you'd like to share.

This is not the actual text that we would use, this is something off the top of my head that captures the key points that we'd like to convey.

But, there's problems with this ...

It could be a year before you actually graduate

Good designers are very difficult to find. If we found 10 great designers, we'd hire all 10 of them on the spot. They have to work on whatever is strategically important for the company at any given time, most of the time that's going to be getting new graduated sites launched - but we have careers, advertising and internal products that need them too.

Barring the discussion that we'd open at the end, we're basically coming to you and saying "We think you're awesome, and good things will eventually happen for you." - that's not much different than anything else we say throughout the process.

While it's intended to tell you that you've crossed a metaphorical finish line, it's also telling you that you have to wait months or longer to get your ribbon. That's less than ideal, and it turns what's currently a major event into something sort of 'meh'.

They got their message when [stats], why haven't we gotten one yet?

Even before coming to work here, I've urged folks to stop looking at those stats so much - while they do allow us to see when a site has consistently good and climbing levels of participation, they aren't the only thing that we look at. This is not an exhaustive list:

  • Are there unresolved concerns around content policy, or something else that could cause the site to shift course?
  • Are there any quality issues that concern us?
  • Do new users tend to stick around?
  • How long does it take for stuff to get answered on average?
  • Is a noticeable percentage of the content basically copied (with attribution) from Wikipedia or other places?
  • Have we created something better than most other sites surrounding this topic?
  • Can they sustain an election?
  • Can they sustain a shift in privilege thresholds?

... and possibly 50 more things that come up during the course of evaluating the site, it really depends on the topic. Those conversations happen among us at first, because it's hard enough to conclude such a discussion with 10 involved, opening that up to a few thousand would be ... a mess (it's purely for practical reasons).

New sites do better when they focus on what they're doing well. We may have concerns, and notice that folks on meta share the same concerns and are actively working on them. There'd be little point in us coming to you with what you already know.

If there are any serious concerns that come out of an evaluation and we don't see you talking about them, we do bring them to your meta site, usually the same or next day. But we're not going to frame that as "Y'all scared off all the noobs, so we're punishing you by not letting you graduate" and if we even mentioned that these concerns came out of a graduation review, many would see it like that.

Put simply, the absence of such communication from us after a period of time is very likely to cause unnecessary angst. Some sites have remained in beta for several years.

Our current system isn't optimal, at all.

A few on the team are diving pretty deep into historical data to find correlations of characteristics of sites as they were when we graduated them. This is in no way an attempt to automate the process of graduating sites based on data alone, but more of a surety that we don't miss sites that might be ready, where 'ready' is only apparent with historical context.

Remember me previously mentioning 10 people talking about these? That's also a problem, it's nearly impossible to get 10 people to agree on anything - so there's a sort of majority system. The problem is, these discussions can inject trepidation for some that wasn't there previously - and part of that is good - humans are prone to oversight. But it's not optimal, and until we make it optimal, I think it would be a mistake for us to suck you into what is currently a rather crazy process.

tl;dr;

We'd like to do this, but it's first going to require a pretty large revamp of how we approach the process, because we're going to be bringing you into that process much sooner than later by doing so, and I'm not certain that's advantageous given how it currently works.

We want to arrive at a system where graduation is less opaque, and that's one of the things we're going to be talking about a week from now when everyone is in Denver.

Note, I'm not saying yes, no, maybe - I'm just letting you know where things are internally.

  • Note, there are other things we're thinking about, it's impossible to capture all of it in a single answer. 'Graduating' without a design is also something we've considered, but that's an entirely different rabbit hole. It's complicated, but we're pretty focused on making it better. – Tim Post Oct 15 '14 at 5:04
  • If design is a key bottleneck, have you considered putting more onus on each community, eg if they can pull together a design that's 90% of the way there can that speed things up. I guarantee communities will find a way. This could reduce a lot of the load on you. They may not be professional designers (or they may!) but lets use the people who are passionate about the community to help. Benefit for the community is shorter time to graduation. – Ro-Reinstate Monic-Alsop Oct 15 '14 at 9:57
  • 1
    @RoryAlsop That sounds pretty bad, to be honest. I don't mean it as a slight against folks, but there's a reason we hold out for an available designer. For one, a good design requires a lot of familiarity with how our sites are set up on the technical side. For another, someone somewhere will still have to convert community ideas into assets, CSS, and so on. The workload on us isn't going to be at all diminished, and in some cases may well increase. Not to mention designer frustration from just executing instead of creating. – Adam Lear Oct 15 '14 at 21:28
  • Ok. That makes sense. It was a suggestion in the dark as i really have no visibility of what happens behind the scenes. Thanks. – Ro-Reinstate Monic-Alsop Oct 15 '14 at 21:30
  • 2
    @RoryAlsop It's just too much of an ask for unpaid volunteers, because it's something we'd have to be extremely picky about. It did cross my mind - It's not that we couldn't do it, it would just be a really bad experience. The saving grace is, I have a bunch of sheep, expensive-looking lab equipment and a plan for no one to notice Jin gone for a few weeks ... – Tim Post Oct 15 '14 at 23:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .