Over two years ago now, we made a big shift in how we handle "graduation" - nearly severing the connection between "graduation" and site design. We also talked about removing the "beta" label at some point before sites “graduated” - when sites are clearly past that phase but don't yet qualify for "graduation" under our current rules. I'd like to propose a scheme that doesn't use the term "graduation" at all, doesn’t keep a “beta” label on stable sites for years and years, and lets all of our sites flourish, without over-burdening the community managers and other staff.
It’s a four-phase process:
- Phase 1 - Private/public beta - unchanged.
- Phase 2 - Site - includes setting up migration paths (if needed) and community advertisements. Also, a single new site design shared with all non-beta, non-custom-designed sites.
- Phase 3 - Full site with elections - includes increased reputation levels, moderator elections.
- Phase 4 - Custom site design - as allowed by designer time availability, includes swag.
It’s easy for me to say this but to explain why… is complicated… and to define these levels further and what should trigger movement between the stages is even moreso.
This started with recognizing that Stack Exchange has a problem. We have 70 sites that have existed on this network for over two years and are yet still considered “beta” sites - some upwards of seven years (Board & Card Games, Homebrewing, Sound Design, and Writing). Some of these sites have many more questions than “graduated” sites, more traffic, more answers… and yet, because they failed to meet the goal for “graduation” they’ve been stuck in a perpetual “beta” state.
This is demoralizing.
It may be OK for Google Mail to be in beta for five years but… why a Stack Exchange site? Do sites like Parenting or Gardening & Landscaping really still have stuff to iron out that they haven’t run across in six years? Even if they do, is that unusual? Meta participation doesn’t cease the moment a site “graduates”... the site continues to grow and change over time.
But this, I propose, isn’t the problem, it’s a symptom of the actual problem.
There's a difference between when a site is no longer "beta" and when a site is ready to be considered "graduated" For years, the network has determined which sites “graduate” based on which sites reach ten questions per day consistently.
Pops looked at a wide variety of statistics and how they tracked on sites we've graduated in the past, and discovered that there was a strong correlation between graduation and the number of new questions a site receives per day. Fast forward through all the boring analysis, and here's the result: from now on, when a site starts to consistently receive ten new questions every day, we'll consider it for graduation. This is not 100% automated; CMs will still be manually checking on how individual communities are doing, and some sites might still graduate "earlier" or "later" than their question activity alone would suggest; but it's an effective rule of thumb.
If “beta” means the site is still “in development” how does that relate to number of questions per day in an absolute way?
I argue that it doesn't. Time, meta participation, and user engagement do that.
- Is there consensus on the site for what’s on topic and what is not?
- Does the site have custom close reasons?
- Do the users vote to close off-topic questions (and participate in other moderation activities)?
- Have they completed their help pages?
- Are there FAQs that help users improve their question and answer quality?
- Is there a reasonable tagging policy in place?
These and other similar factors are what help a site decide what it needs to work on while in the “beta” phase and what should be figured out before a site "graduates". They're discussed here on Meta Stack Exchange in the FAQ on Beta sites and in the old blog post linked to in the FAQ.
If these are the goals that we set for ourselves as beta sites, why are we using questions per day to decide when a site is no longer in beta?
Firstly, we aren’t. Because we currently only have two states, “beta” and “graduated”, a huge number of sites don’t meet the “graduation” requirements but are well past managing the tasks put to them as their work in the “beta” stage. This proposal introduces that median state. I'm going to say that, while it may be a good indicator of when to "graduate" a site, ten questions per day may not be the indicator we should be using to determine when the "beta" phase should end.
Two and a half years ago, “graduating” a site was onerous. It required a ton of work on the part of the developers, community managers, and design team. The “graduation” model makes their work more complicated for every new site “graduating” - so maybe we should fix that.
With the old “graduation” process, after a site went into public beta, there was one step - “graduation” with a custom site design. In September of 2015 we split these in two, first removing the "beta" label and holding elections and then, later, rolling out the new site design and privilege changes. Holding off on site design freed up the design team from being the sticking point for “graduating” a site. I recommend we further adjust this to allow more sites to have the “beta” label removed without requiring the more onerous work of running elections to occur. This leaves us with a four-phase process, as outlined briefly above and, in more detail, here...
How long is public beta? - Phase 1 & When does it end? - Phase 2
Robert Cartaino has always championed the concept of a public beta only lasting about ninety days. I don’t think this is enough time on some sites… possibly many. I’m currently a moderator on a site that’s been in beta for about 120 days and we’re nowhere near ready to be out of “beta”. But let's not focus on a magic number; every site is different. Instead of having CMs check in on some schedule, let the sites decide when to ask for the change.
Most of this work should fall on the the users and moderators. They are in charge of meeting the beta requirements. Much like requesting tags to be blacklisted or proposing site-specific features, when they feel that they meet the requirements for leaving beta, the moderators can ask the Team to review their request in the form of a meta post explaining why they’re ready to have the beta label removed. If the Team agrees, a few things happen:
- The “beta” label is removed.
- The site gets a new site theme, shared by all Phase 2 (and 3) sites. This lets the entire Network see that they’ve moved beyond the beta phase and are a “full site”.
- The site gets the ability to set up migration paths.
- The site gets the ability to have community ads.
If the Team disagrees, they should respond to the meta post explaining why.
Why the new site theme? Isn’t this more work for the design team?
Look at these sites. In this image, there are six "graduated" sites without custom designs.
Can you tell which are graduated sites with beta-style logos and which are beta sites? Click the image to see the answer.
Changing the site theme and logos will make it obvious which sites are still beta and which are full sites, especially if there are dozens of them. This gives users something to strive for and sets the sites apart more than the meager removal of the word “beta”. This is further discussed very well by Monica Cellio in her answer to my question Give Graduated sites waiting on a custom design something to set themselves apart from Beta sites.
Why migration paths and community ads?
These are, I think, things that are pretty easy for the Team to implement. Migration paths (when needed) are one-time changes, and the community ads post is a once-yearly bit of work, not something done throughout the year. They give the site something more (in addition to the non-beta site design) and recognize that they’re full sites by allowing them to stand as peers with the other full sites on the network.
When do we elect moderators? - Phase 3
Just like today, elect moderators when there’s enough activity on the site - ten questions per day - and when there are sufficient active high-reputation users to run the site adequately with the higher reputation levels. I recommend moving privilege changes to this phase because higher reputation levels shouldn’t be tied to the design team’s prioritization of when to work on Anime’s site design. These two things are irrelevant of each other.
How does increased questions per day imply that the site is ready for elected moderators and increased privilege levels? If we assume there's a good voting culture on the site, when more questions are entering the site's ecosystem regularly, more answers can be written. That (hopefully) means there's more reputation to go around and more users able to achieve these higher reputation levels required for the upper level privileges.
But… what about the custom site designs… and swag? - Phase 4
It’s become clear to me that the Stack Exchange Network has higher priority work for their design team than making beautiful site designs for our many “graduated” sites… and that’s OK. It doesn’t mean that they will never design another site (hopefully) but it does mean that we should probably work towards letting them off the hook further by allowing them do the designs in their own time rather than tying some part of the “graduation” process to their work. As such, Phase 4 is aspirational. Sites that have achieved Phase 3 status are eligible for custom designs but there’s no guarantee when or if they will happen. This is why I strongly recommend we ask the design team for a single new site design for all Phase 2/3 sites.
Swag depends on the sites having a design. There’s no way around this. That means that the big celebration for the top users, in addition to the design, is getting cool stuff from Stack Exchange.
What would the effect of this be?
If my count is correct, there are currently ten sites in what would become Phase 3 that have had elections but have not had the increased privilege levels. Some of these sites probably need to wait a bit for the increases (Data Science, Law, Arduino) but most of them should be able to manage it right now.
The bigger change is that all of the 70 sites that have been in beta for two or more years (and possibly some/many of the sites between a year and two years) can move to Phase 2 upon the users and moderators discussing it as described above. There may be a few of them that want to hold off for a bit if they’re dealing with some major scope considerations - Psychology & Neuroscience and Health, for example. But this would push the bulk of the “beta” sites out from under the “beta” label quite quickly, though the CMs can probably transition them over gradually as time permits.
For nearly a decade, users have come together to build high-quality communities on Stack Exchange. Isn't it time to recognize that work, to end the demoralization that comes from being shackled to the "beta" label long past the time it was an accurate description? Let’s make this our goal for 2018.