Let's break up with "Graduation" and remove a bunch of "Beta" labels

Question

Over two years ago now, we made a big shift in how we handle "graduation" - nearly severing the connection between "graduation" and site design. We also talked about removing the "beta" label at some point before sites “graduated” - when sites are clearly past that phase but don't yet qualify for "graduation" under our current rules. I'd like to propose a scheme that doesn't use the term "graduation" at all, doesn’t keep a “beta” label on stable sites for years and years, and lets all of our sites flourish, without over-burdening the community managers and other staff.

It’s a four-phase process:

• Phase 1 - Private/public beta - unchanged.
• Phase 2 - Site - includes setting up migration paths (if needed) and community advertisements. Also, a single new site design shared with all non-beta, non-custom-designed sites.
• Phase 3 - Full site with elections - includes increased reputation levels, moderator elections.
• Phase 4 - Custom site design - as allowed by designer time availability, includes swag.

It’s easy for me to say this but to explain why… is complicated… and to define these levels further and what should trigger movement between the stages is even moreso.

The Symptom

This started with recognizing that Stack Exchange has a problem. We have 70 sites that have existed on this network for over two years and are yet still considered “beta” sites - some upwards of seven years (Board & Card Games, Homebrewing, Sound Design, and Writing). Some of these sites have many more questions than “graduated” sites, more traffic, more answers… and yet, because they failed to meet the goal for “graduation” they’ve been stuck in a perpetual “beta” state.

This is demoralizing.

It may be OK for Google Mail to be in beta for five years but… why a Stack Exchange site? Do sites like Parenting or Gardening & Landscaping really still have stuff to iron out that they haven’t run across in six years? Even if they do, is that unusual? Meta participation doesn’t cease the moment a site “graduates”... the site continues to grow and change over time.

But this, I propose, isn’t the problem, it’s a symptom of the actual problem.

The Problem

There's a difference between when a site is no longer "beta" and when a site is ready to be considered "graduated" For years, the network has determined which sites “graduate” based on which sites reach ten questions per day consistently.

Pops looked at a wide variety of statistics and how they tracked on sites we've graduated in the past, and discovered that there was a strong correlation between graduation and the number of new questions a site receives per day. Fast forward through all the boring analysis, and here's the result: from now on, when a site starts to consistently receive ten new questions every day, we'll consider it for graduation. This is not 100% automated; CMs will still be manually checking on how individual communities are doing, and some sites might still graduate "earlier" or "later" than their question activity alone would suggest; but it's an effective rule of thumb.

If “beta” means the site is still “in development” how does that relate to number of questions per day in an absolute way?

I argue that it doesn't. Time, meta participation, and user engagement do that.

• Is there consensus on the site for what’s on topic and what is not?
• Does the site have custom close reasons?
• Do the users vote to close off-topic questions (and participate in other moderation activities)?
• Have they completed their help pages?
• Are there FAQs that help users improve their question and answer quality?
• Is there a reasonable tagging policy in place?

These and other similar factors are what help a site decide what it needs to work on while in the “beta” phase and what should be figured out before a site "graduates". They're discussed here on Meta Stack Exchange in the FAQ on Beta sites and in the old blog post linked to in the FAQ.

If these are the goals that we set for ourselves as beta sites, why are we using questions per day to decide when a site is no longer in beta?

Firstly, we aren’t. Because we currently only have two states, “beta” and “graduated”, a huge number of sites don’t meet the “graduation” requirements but are well past managing the tasks put to them as their work in the “beta” stage. This proposal introduces that median state. I'm going to say that, while it may be a good indicator of when to "graduate" a site, ten questions per day may not be the indicator we should be using to determine when the "beta" phase should end.

What next?

Two and a half years ago, “graduating” a site was onerous. It required a ton of work on the part of the developers, community managers, and design team. The “graduation” model makes their work more complicated for every new site “graduating” - so maybe we should fix that.

With the old “graduation” process, after a site went into public beta, there was one step - “graduation” with a custom site design. In September of 2015 we split these in two, first removing the "beta" label and holding elections and then, later, rolling out the new site design and privilege changes. Holding off on site design freed up the design team from being the sticking point for “graduating” a site. I recommend we further adjust this to allow more sites to have the “beta” label removed without requiring the more onerous work of running elections to occur. This leaves us with a four-phase process, as outlined briefly above and, in more detail, here...

How long is public beta? - Phase 1 & When does it end? - Phase 2

Robert Cartaino has always championed the concept of a public beta only lasting about ninety days. I don’t think this is enough time on some sites… possibly many. I’m currently a moderator on a site that’s been in beta for about 120 days and we’re nowhere near ready to be out of “beta”. But let's not focus on a magic number; every site is different. Instead of having CMs check in on some schedule, let the sites decide when to ask for the change.

Most of this work should fall on the the users and moderators. They are in charge of meeting the beta requirements. Much like requesting tags to be blacklisted or proposing site-specific features, when they feel that they meet the requirements for leaving beta, the moderators can ask the Team to review their request in the form of a meta post explaining why they’re ready to have the beta label removed. If the Team agrees, a few things happen:

• The “beta” label is removed.
• The site gets a new site theme, shared by all Phase 2 (and 3) sites. This lets the entire Network see that they’ve moved beyond the beta phase and are a “full site”.
• The site gets the ability to set up migration paths.
• The site gets the ability to have community ads.

If the Team disagrees, they should respond to the meta post explaining why.

Why the new site theme? Isn’t this more work for the design team?

Look at these sites. In this image, there are six "graduated" sites without custom designs.

Can you tell which are graduated sites with beta-style logos and which are beta sites? Click the image to see the answer.

Changing the site theme and logos will make it obvious which sites are still beta and which are full sites, especially if there are dozens of them. This gives users something to strive for and sets the sites apart more than the meager removal of the word “beta”. This is further discussed very well by Monica Cellio in her answer to my question Give Graduated sites waiting on a custom design something to set themselves apart from Beta sites.

Why migration paths and community ads?

These are, I think, things that are pretty easy for the Team to implement. Migration paths (when needed) are one-time changes, and the community ads post is a once-yearly bit of work, not something done throughout the year. They give the site something more (in addition to the non-beta site design) and recognize that they’re full sites by allowing them to stand as peers with the other full sites on the network.

When do we elect moderators? - Phase 3

Just like today, elect moderators when there’s enough activity on the site - ten questions per day - and when there are sufficient active high-reputation users to run the site adequately with the higher reputation levels. I recommend moving privilege changes to this phase because higher reputation levels shouldn’t be tied to the design team’s prioritization of when to work on Anime’s site design. These two things are irrelevant of each other.

How does increased questions per day imply that the site is ready for elected moderators and increased privilege levels? If we assume there's a good voting culture on the site, when more questions are entering the site's ecosystem regularly, more answers can be written. That (hopefully) means there's more reputation to go around and more users able to achieve these higher reputation levels required for the upper level privileges.

But… what about the custom site designs… and swag? - Phase 4

It’s become clear to me that the Stack Exchange Network has higher priority work for their design team than making beautiful site designs for our many “graduated” sites… and that’s OK. It doesn’t mean that they will never design another site (hopefully) but it does mean that we should probably work towards letting them off the hook further by allowing them do the designs in their own time rather than tying some part of the “graduation” process to their work. As such, Phase 4 is aspirational. Sites that have achieved Phase 3 status are eligible for custom designs but there’s no guarantee when or if they will happen. This is why I strongly recommend we ask the design team for a single new site design for all Phase 2/3 sites.

Swag depends on the sites having a design. There’s no way around this. That means that the big celebration for the top users, in addition to the design, is getting cool stuff from Stack Exchange.

What would the effect of this be?

If my count is correct, there are currently ten sites in what would become Phase 3 that have had elections but have not had the increased privilege levels. Some of these sites probably need to wait a bit for the increases (Data Science, Law, Arduino) but most of them should be able to manage it right now.

The bigger change is that all of the 70 sites that have been in beta for two or more years (and possibly some/many of the sites between a year and two years) can move to Phase 2 upon the users and moderators discussing it as described above. There may be a few of them that want to hold off for a bit if they’re dealing with some major scope considerations - Psychology & Neuroscience and Health, for example. But this would push the bulk of the “beta” sites out from under the “beta” label quite quickly, though the CMs can probably transition them over gradually as time permits.

For nearly a decade, users have come together to build high-quality communities on Stack Exchange. Isn't it time to recognize that work, to end the demoralization that comes from being shackled to the "beta" label long past the time it was an accurate description? Let’s make this our goal for 2018.

Answer 1

I'm speaking from the perspective of someone who uses and moderates several low-activity beta sites. I agree that it can be, at times, a bit demoralizing to see some sites quickly graduate,¹ while others of us get left in the dust. At times, I feel a bit like Spirit, waiting for that call from NASA - except I know that there's no time at which I should expect my site to graduate.²

Let me take Astronomy, a site I think I know and understand pretty well, as an example. The Area 51 stats are a mixed bag - low questions per day (3.2), a decent percent answered (93%), a strong userbase, and decent views per day (3,150). I've been on Astronomy for three years now, roughly 75% of its age, and I've seen those stats remain the same. I've kind of resigned myself to the fact that they may never change. But there are some things that, once in a while, make me think that there's hope for graduation, or some sort of progression:

• We don't have any huge scope issues; those have been dealt with. Our boundaries with some other sites are fuzzy, yeah, but if a question's on-topic here, we'll keep it.
• We have a consistently populated community. People have come and gone, as people do, but that's not really a problem, because others have filled their shoes. A community should outlive its founders; if not, it's not at all healthy.
• Stuff that shouldn't stay around generally gets deleted or closed, as it should. We do have the caretakers that Ana talked about two and a half years ago as an important criterion for site survival.

I'm not trying to make a case for Astronomy to graduate; I'm just using it as an example because I know it really well. There are other sites I know not quite as well, but which I suspect have the same stats or better, and are also strong in the community, moderation and scope areas. Writers and History come to mind. The sites fail the 10 questions per day requirement, but they're really strong everywhere else. And they're getting good traffic³ - 10,000+ views per day for these last two - meaning that they're helping people, in another way.

So as, like Spirit, we the people of the small betas trudge on through the wasteland of Mars day-to-day existence, we sometimes wonder what we're aiming for. Is content rate all that we need to improve? Is that all we're doing wrong? We might not reach 10 questions per day within the next month, or year, or decade, but we're doing pretty darn well everywhere else. In fact, you might say that . . .

. . . we're stable. We're not stably producing 10 questions per day, but we're stable nonetheless. We're stable at 3, or 4, or 5, or 6 questions per day. I'm not gonna argue against the data that says that 10 questions per day indicates a stable site producing a healthy content . . . but I will argue that these sites have passed the initial trial. We can do it. We can maintain a level of questions per day for years on end, even as people come and go. I'd say that's pretty successful.

I'm going to state a claim. I hope it's wrong, but what I've seen fits the data: Not every topic is going to beget a site that gets 10 questions per day within any reasonable amount of time. Writers has been in beta for seven years, and it's not seeing a huge rise in question rate. And yet the site goes on, and it seems to still be doing okay, and it's stable (yes, it's stable!). But nobody would say that it should be closed down, right?

Let's say you have a niche topic - like, say the history of science and mathematics. There aren't a lot of people interested in that, right? Certainly not as many as are interested in the actual science and mathematics involved. To reach those 10 questions per day might mean that you need to get more people interested in the topic - and that could be a good thing. But it seems a little unreasonable to expect a site to do that to graduate. Satisfying the current people interested in the topic is a lot more important. If the people of professionally study the history of science and mathematics, and the people who read about it as enthusiasts, are happy with the site as a great resource to go to to ask questions and get answers, isn't that successful?

That site might be reaching a much smaller fraction of the humans on Earth than, say, [example site that's metaphorically on fire]. But it could be a healthier site if it's met all those other metrics. I'd rather have a low-traffic site that's healthy than a high-traffic site that still has some problems. Obviously, question rate is far, far from the only consideration for graduation. But it shouldn't be what holds a site back.

Time and again, we the people of the beta sites, thinking "Hey, this is what we really need to work on, and if we raise the question rate, maybe we'll graduate!", make a push to try to raise question rates, such as topic challenges. Sometimes they work for a bit - a week, a month - but many times, they eventually collapse, and we go back to our equilibrium question rate. And then we're left still stuck on Mars, wondering what we did wrong. I can say from experience that this sort of thing really, really stinks.

Let me say what I like, in particular, about this proposal:

• It gives some things to work for in the early days of each site. After private beta, a lot of people often leave a site. I've been in this group a lot. Sometimes, it's because I feel like I couldn't really contribute much to a site (Open Source was an example for me). Other times, life outside the Internet pops in, or I find that I need to put more time back into other sites. But for many cases, I just can't summon the urge to devote myself to a site that's going to be left in limbo for . . . how long?

  Maybe this is partly the immaturity and impatience of youth. Could be. I'm willing to own up to my faults. But it's not hard be demoralized when only one goal in site - and that's far off. That's what this proposal largely talks about. This is what I like about Phase 2: It's an acknowledgement that a site's solved some problems, and gotten through some growing pains. Did we solve scope issues? Have we gotten our close reasons straight? Do we have systems in place to guide new users (the help center, FAQs, etc.)? If so, it would be nice to have an official acknowledgement. If we don't hit 10 questions per day . . . we don't get that. Our work feels a bit unrecognized.

• I also like that the proposal breaks up the process into related chunks. I feel like there's a growing popularity of this sort of step-by-step option for beta sites - starting, or at least becoming more popular, after Ana's meta post. In this plan, the changes to the site come as needed, and make sense. Community ads and migration paths⁴ come as expected: when the scope's straightened out. When we do reach those golden 10 questions per day - and yes, they still have a role in this plan - we get the upgrades that come along with higher traffic (elected moderators, raised privilege levels, etc.). I also agree with keeping the site design for the very end, as a sort of ultimate reward.

• I also like that it involves the users and moderators taking the lead when it comes to figuring out if the site's ready. Catija suggested

  When they feel that they meet them, the moderators can ask the Team to review their request in the form of a meta post explaining why they’re ready to have the beta label removed.

  As a mod, I feel like I can always talk with a CM about something if I need to. Y'all do a fantastic job of being accessible to us mods. But the dialogue with the userbase as a whole sometimes isn't as great as it could be. In part, that's on us users. I picture a bunch of penguins standing around looking at each other and being a bit clueless about our site's health. Having an outside opinion from some of the best in the business helps a lot, obviously, and that's what currently decides whether a site's fit to graduate - and will still do so, in this model.

  But the decision becomes that of the userbase. It forces us to do our own self-evaluation,⁵ look at our problems and figure out if we've fixed them or not. That's . . . great. We're no longer Spirit; we have to actively and formally figure out if we think we can graduate. And if a community doesn't really care enough . . . then it won't necessarily graduate. And that's fair.

So, to summarize this long and jumbled journey of an answer:

• A lot of sites seem to mostly have their stuff together, and are sitting at low activity levels . . . but they're certainly stable.
• I'm betting that many will never end up reaching those levels of activity we all sort of aspire to.
• This proposal takes emphasis off of that criterion - although it does not remove it.
• Having these intermediate benchmarks is encouraging, as a user.
• These stages, and the process around them, seems like it might encourage discourse and discussion on growth issues by the community.

Maybe this is an optimistic vision of it all. It probably is. But hey, on Astronomy and Writers and History and so many more, it feels like we're stranded on Mars. This is a way off that planet, one that challenges and encourages us to build a better site and strive for goals, without setting unreasonable ones.

---

^{1 I've used, and now moderate, some of these sites, so don't think I'm jealous! Sites that graduate deserve celebration and some recognition.
2 The 90 days reference in that comic is coincidentally equal to the original 90 days (now since deprecated) of a beta site.
3 I think I'm an outlier in Stack Exchange, someone who came in from these small to middling sites, so my expectations for activity aren't, and never have been, comparable to those of some of the larger sites. So I have a different perspective than many folks.
4 I'm not a huge fan of the migration system, but it's here for now. That's a whole 'nother discussion.
5 The old ones are mostly gone, as of two years ago.}

Answer 2

I've been excited to respond to this question because it gets at one of the biggest deficiencies I see on the network. After the Area 51 process, there are precious few community achievements. Graduation stands as the one thing a site can look forward to after passing through private beta. Traditionally, graduation implied a commitment to keep the site on the network which was sealed with an investment in a professional design. More recently, we started graduating sites without also giving a design. This was mostly a stop-gap solution since our design team couldn't keep up with sites passing through our newly-minted heuristic.

Personally (and I should make clear this post is my opinion and not the conclusion of the Community team, though I bet we are mostly on the same page) I felt this was one step in the direction of giving sites bits of graduation as needed. But the trouble with separating design from the functional parts of graduation is that it's not easy to tell what stage a particular site might be in. You, as a loyal reader of meta, know that there's a difference between Code Golf and Biblical Hermeneutics because one has the word "beta" after it and the other doesn't. With some detective work, you could figure out what other aspects of graduation Code Golf has, but the average user would not even know most of those things exist.

One of the reasons I haven't responded to this question before now is that the symbols I thought of would be either too expensive (meaning, hiring more designers) or too impersonal. Removing the beta label might mean we are committed to your site, but it's not a very good symbol. It's a bit like having a wedding where the couple doesn't exchange rings, take picture or have a party afterward. Those symbols mean something. (And if your culture doesn't have wedding rings, I'd be willing to bet you have some other set of symbols that signal commitment.) Wedding rings work not because they are costly (though they usually are) but because they represent the specific relationship between two people.

This week we got a potential solution: standardized site themes. Obviously the priority will be to get sites with existing designs converted to customized themes. Because of loss aversion, I've been focusing on how those sites would become less unique. But the next step will be to start building designs for sites that have been patiently (or not so patiently) waiting for them. Because theming reduces the range of design elements to consider, that process should go much faster. Assuming that's true, custom design elements would be good symbols of our commitment to a site.

Graduation, of course, would be signaled with a full design visible to all visitors. Beta sites (however we define that) would retain the beta theme. In between, sites can accumulate incremental design changes. Perhaps the first order of business after a site exits beta is to decide on a color scheme. That way visitors will see at a glance that the site has a achieved something significant.

Other design elements like a background image and a logo could be tied to moderator elections and privilege level changes. These are somewhat weaker signals and it might paint designers into a corners, so to speak. So maybe we shouldn't try to be too rigid here.

Answer 3

I think it would be better to start with changing the beta header with a custom image, then to follow the four phases you listed.

Why?

Because I am a regular user of Gardening and Landscaping SE and from what I can tell the site has two options of reaching 10 questions per day or else to stay in beta forever:

Option 1: Convince much more people from Australia and New Zealand to use the site in order to counteract the low traffic coming in winter fron Northern hemisphere. Is this a doable approach?

Option 2: Convince more people worldwide to use the site. To achieve that, we need to be seen as an alternative to a dozen of other established sites that have at least a tree, a bunch of green leaves or some colorful flowers on their front page.

I dare anyone to give me a link to a site with a similar scope and more questions per day that doesn't have a plant somewhere on their home page. For people in the gardening habit, content means plants not a block of text covering 100% of the front page.

I will stick to GL no matter what, beta or not beta, tree or dull front page, but how many people do you know to not care about the appearance of a site?

I want to stress that I'm suggesting just an image on that light blue header, not full design.

Answer 4

While I like the idea of handing out a Phase 2 design and agree with most of your points regarding the advantages of that approach that doesn't address one very important thing the sites with the full design really have on beta sites. Site recognition.

That's why I think Alina's idea which I gladly up voted is a better first step towards moving beta pages forward. Giving beta sites which meet certain criteria—which might be as simple as making it half a year or a year without dying off—a site logo of their own. A logo that is in the overview, the tabs and the header of the site.

This might even be a preliminary logo and it's just a logo. Something that gives the site some better identification and recognition. Those two are things that very likely improve the sites chances to reach the other metrics to actually graduate, reach the next phase or however the process might look like then.

That single logo (yeah I know in some different dimensions and stuff) might not even have to be designed by the SE design team. Maybe the site's meta can come up with it. Maybe the 75k users of the Graphic Design SE can lend some expertise (or hold contests). If there were some guidelines about those preliminary logos I'd think the community could come up with great creations to give the more stable beta sites some better identification and brand recognition.