Graduation, site closure, and a clearer outlook on the health of SE sites

Question

Back in April of 2010, Joel shared our assumptions about the role of small sites in the newly minted Stack Exchange network:

If a site does not have enough activity at the end of 90 days, it will be closed down. Any existing Q&A will be archived and made available for download, but the site itself will not remain live. Small, unhealthy sites do nothing but draw traffic away from other sites, splitting audiences, so we don’t want to keep them around.

Past experience had led us to believe that “small” and “unhealthy” would mean the same thing: we took it as a given that quiet sites would develop spam and obvious broken windows, while high quality sites would always grow big and graduate. But you proved us wrong! Five years later, we have lots of tiny sites which have been in public beta for months or years, each consistently producing excellent Q&A which helps people with real problems. Small, consistently active sites are great!

Our old assumptions didn’t pan out, but we never provided clear updated criteria for when we would close or graduate a Q&A site. We're rectifying that now.

The TL;DR:

1. When a site starts to consistently receive 10 questions/day, we’ll consider it for graduation.
2. If a public beta site does not produce consistently helpful content, and lacks the caretakers needed for flags and spam to get handled and our Be Nice policy to be upheld, it will be closed.

Is your site in between these two categories? You don’t have anything to worry about. Regardless of how small the site might be, you have a home here in the SE network. Spoiler alert: As of this post, no site which is currently active is at risk of being shut down.

If you’d like to learn more about the nitty gritty details, read on.

---

Clarifying graduation

While Community Managers have historically had high levels of agreement about when a site should graduate, we arrived at our conclusions via totally different methods. When Pops joined the team after being a longtime user, he noticed this, and decided to examine our assumptions and procedures. He also wanted to apply some lessons from the research showing that well-designed data-backed algorithms and heuristics are more effective at making decisions than human intuition alone.

Graduation is a hard concept to pin down quantitatively, but we basically expect that a graduated site should be able to continue growing and governing itself indefinitely without any outside help unless something truly catastrophic happens.

Pops looked at a wide variety of statistics and how they tracked on sites we've graduated in the past, and discovered that there was a strong correlation between graduation and the number of new questions a site receives per day. Fast forward through all the boring analysis, and here's the result: from now on, when a site starts to consistently receive ten new questions every day, we'll consider it for graduation. This is not 100% automated; CMs will still be manually checking on how individual communities are doing, and some sites might still graduate "earlier" or "later" than their question activity alone would suggest; but it's an effective rule of thumb.

Tangible guidelines for site closure (aka Reasons why you shouldn’t be afraid)

While Pops worked to clarify graduation, I looked at things from the opposite direction: site closure. The precedent was set early on that sites had two options; graduate, or get shut down. This created a lot of unnecessary angst for our smaller and mid-sized communities. Lots of folks on excellent sites assumed that since they hadn’t cleared the graduation hurdle yet, we might come in and turn off the lights at anytime. Not exactly great for morale.

Thanks to many devoted users, it’s grown clear that smaller SE sites can do a great job of maintaining themselves and producing high quality Q&A. Not every site is going to be a blockbuster success, but our small sites are serving their own communities well. We’re proud of you, and we want you here.

What does this mean? If there's enough moderation for a public beta site to consistently remain free of spam, for flags to be cleared, and for our Code of Conduct to be upheld, your site will remain open. However, if community leaders drop off, flags sit without being addressed, and we can’t find any suitable volunteers to step forward, the site gets closed.

As of this post, not a single site currently active in our network is at risk of being closed. Closing public beta sites is a rare occurrence; we expect it to stay that way.

Success and graduation are not the same thing

When the network was just getting started, we all worked off of judgment calls and personal experience because that was all we had. Since then, we've tested our assumptions and we’re excited to have been able to create more objective criteria for some of our most important decisions. The "trial by fire" approach to site launches and graduations was a great early motivator for communities, but in reality, success and graduation aren’t the same thing. Now, communities can focus on Q&A about topics they love, not on whether their site will survive. We’re proud of what you’ve accomplished so far. We hope you are too.

Answer 1

As you seem to be renaming stages anyway, how about uncoupling the different aspects of graduation, i.e., do not enforce them to coincide? As it seems to me right now, there are six effects of graduation:

• Site Design: This is obviously causing most of the work and it’s reasonable that you only want to do this, if a site has sufficient activity and survival chances to justify the effort. So, this is one thing that should certainly only happen at 10 QPD.

• Moderator Elections: This does not seem to be causing much work (as it is mostly automatised) and could happen whenever there is need for another moderator, be it because a pro-tempore moderator stepped down or due to increasing site activity – and if certain conditions are met (site age, number of users who can vote), which do not necessarily have to be the graduation criteria.

  It seems that you are open to the idea to hold moderator elections without full graduation anyway ([1], [2]), though on sites which are scheduled for graduation anyway. (I begin to suspect that lacking a moderator becomes a possible way to accelerate graduation, once you are in the queue.)

• Community Ads: I do not see any reason why those should not be enabled on any site older than a certain age (so that the direction of the site is clear).

• Reputation Levels: The most important effect of these is that it decides who can review. Thus a large community is required for those levels to be adjusted. Maybe this coincides nicely with graduation, maybe a different criterion is appropriate – I would need more data and experience to decide this.

• Full migration target: Is this still a policy? I have seen some migrations to beta sites and most of the time the alternative was outright closing the question. I can see that migrating a borderline question to a site that has just reached public beta and thus may vanish soon is a bad idea – but then borderline questions should probably stay where they were intially posted anyway. Anyway, I do not think that graduation is required for this status. Maybe in some cases it need’s reminding that questions can be on-topic on more than one site.

• Being listed in the footer: I am somewhat indifferent about this.

So for example, how about the following?

• The actual label beta is applied for about a year, with possible extensions if the site is still struggling, but alive.
• Afterwards, certain features of graduation (elections, community ads, full migration target, listed in footer) are enabled and the label beta is removed.
• If the site has reached 10 QPD, the site has won the Internet and gets other features of graduation (individual design, reputation levels).

Note that this is really just an example, I would need more data, experience and insights into SE to tell whether this suggestion is really good.

Answer 2

As an active participant in a low-volume, but high-quality (well, mostly!) SE site, this is reasonably welcome news.

One of the most tangible effects of "graduation" is, of course, the bespoke graphics-css-look-and-feel to reflect the ethos of the site's core interest. "Beta" sites (or whatever they might be called in future -- "niche"? "boutique"?) have pleasant but generic and shared graphics-css-etc.

Given the possibility that a site might stay in beta forever, I wonder what the thinking might be about tossing those sites a bone where some css tweak would help that site's Q&As. The case I have in mind for "my" SE site is a small but significant tweak to the font stack, but I'm fairly confident that other beta-forever sites might have a similar need.

Is it too much to hope that the shift in policy announced by Ana♦ (with Pops♦) might also bring some consideration for small but significant CSS tweaks to the beta-forever sites? "Where there's life, there's hope," as Sancho said to Don Quixote...

Answer 3

On graduated sites the community elects permanent moderators, but on beta sites "pro-tem" moderators are appointed by the SE community managers. This made sense with the original vision of beta -- it'd only be for a year or so, maybe two, and then the community would graduate and choose its own.

With long-running (permanent?) beta sites, you can have moderators who were appointed during the private beta, before the vast majority of the community was present, who remain in the job forever. Sometimes this is just fine -- if a good moderator who remains engaged with the site and serves the community well keeps doing that, we all win. But sometimes, over time, the fit between a pro-tem mod and the site degrades -- a moderator stops coming around as much, lets flags pile up, won't moderate in accordance with community wishes because he personally disagrees with a change in direction, just seems to be a little crispy, whatever. If a moderator behaves in a way that's egregiously bad the community managers might step in, but that's very rare. So things can fester sometimes and that hurts everybody.

One might point out that on graduated sites moderators are also elected "for life" (unless they mess up badly or step down), but there are two important differences. First, on a graduated site there was an established community at the time of the election and that community chose its moderators. That might have been years ago and the community make-up might have changed, but there was some established community making the decision. Second and perhaps more important, beta sites by their nature change over time, and what a community needed on the first day of public beta might not be what it needs three years later. I've been an active participant on several betas, a pro-tem mod on an established one (appointed about two years in), and a pro-tem mod on a brand-new beta. They're different.

With the possibility of appointed moderators serving for many years on evolving sites, I think it serves everybody's interests -- moderators, users, and SE -- to come up with a way to periodically check the community's support of its current moderators -- some sort of vote of confidence for the team. This should allow private response, be solicited (don't just rely on people to click "contact us" if they want to say something), and be done in a way to minimize drama (anonymous survey good, meta voting probably not good). I'm envisioning a poll with two questions: (1) Overall rating for the team (1-5 or whatever), and (2) here, have a textbox.

This isn't an election and the results aren't binding but, as with the site self-evaluations for content, the community managers would use the results of this to help the site in whatever way seems most appropriate -- quietly coach that one mod, add someone new to the team because the workload has gotten high or because a needed skill is lacking, try to contact that absentee mod, tell the mods what they're doing well, nudge someone to start a meta discussion about that theme that came out in a bunch of the comments, etc.

As a user on long-running beta sites I would like us to have the chance to provide this kind of feedback from time to time. As a moderator on such sites I would like to have the chance to receive it.

Answer 4

This is awesome. Thank you.

For quite some time, I've been worried about how a couple of the beta sites I'm active in are doing. This makes me feel like they're just fine, given that they match the criteria you gave. Thank you guys for the overhaul.

However, can the word be spread about this? This question has been getting a lot of attention, and it is, after all, featured, but there are many people who don't check out Meta Stack Exchange (sadly).

My proposal for that is that the Area 51 stats be updated a bit. They can all remain - I find that they're quite nice - but keeping the text as is can make people worry. Seeing the "needs work" on the Views category isn't encouraging - but now that we know that that isn't necessarily a problem, it would be nice if that knowledge was emphasized.

Seeing things in the red is a bit scary. Emphasizing that lower stats are just fine can fix that.

Imported from my comment:

For the "% answered", "users", "answer ratio", and "visits" - and perhaps "questions per day" - I would suggest getting rid of the "needs work" label, and perhaps the "okay" one. At the very least, changing the colors from red would be nice, and a link to this question would be helpful.

Answer 5

This is a step in the right direction

Having just participated in a beta site myself, the process isn't as user-friendly as it should be. I've lost track of how many times people have unrealistic expectations about how the site will grow. I have a few suggestions on how Stack Exchange can do that:

1. Could you change the 7 essential questions link that features on every beta site's meta? It creates unrealistic expectations for new users about how much activity the site will have.

   • Similarly, could you add a disclaimer to old blog posts discussing site closures so it's clear that closing sites solely because of low activity is no longer an SE policy.

2. Could this graph be linked to prominently in new private betas? I think it would go a long way in clearing up unreasonable expectations about beta site activity.

3. Provide more resources to help communities promote their sites. I have no idea what this would look like (maybe just a simple guide on how to effectively use social media and how to invite new users to join the site). If community members have "easy" actionable ways to increase their site's activity, then maybe people will stop asking questions about the site's activity and start doing something about it.

4. As other people have already said, the Area51 stats page should be more user friendly.
5. While I'm at it, a next generation Area 51 would be nice too (though not nearly as important as the first four suggestions).

Thank you for doing something about this, and for taking the time to put many people's minds at rest. Everyone who participates in small beta sites appreciates this.

Answer 6

For sites staying in beta, please consider proposals for Allowing CSS customization of beta sites.

Answer 7

Awesome to hear.

With the new criteria for site closure, is there a chance the previously closed-without-meaningful-warning Answers.OnStartups Stack Exchange could be re-opened?

I know there's a new "Startups" beta that's been opened; it's doing significantly worse compared to the original.

I suspect it's at least partially because so many of us from the original SE site don't care to participate after all our work on the original was unceremoniously trashed.

Giving us our profiles and our content back, merging it into the new site, could bring back a sizable community to what was a valuable site in the SE network.

Answer 8

I think this is a good start. More clarity = happier users, nearly universally.

However, I think that more feedback would be very helpful, particularly on sites that have been around for years and are still in beta. Parenting is my example of this, but I'm sure there are other similar sites: sites that are in beta for many years, and have reasonably active communities, but have one stat they just don't ever seem to get high enough - in Parenting's case, specifically, questions per day.

It would be helpful to have specific feedback at the per-site meta on a regular basis - quarterly, I'd say, if possible - as to how the site's doing and what the long-term plans are. Right now it sounds like permanent beta is where we'll be at Parenting, from the above, and that's probably fine; it sounds like the issue SE has with graduating has to do with picking which sites to spend the (expensive) resources on to give flashy new designs, and I'm at least not particularly concerned with that side of things. But I don't know if that's true, in our site's case. Maybe there is a plan to graduate us at some point. Maybe they'd graduate us if we did X, Y, or Z, that don't specifically involve increasing Q/day. That not knowing can be frustrating at times.

So - thanks for the above details, very much appreciated - but maybe somebody would stop by every so often and talk to us? Thanks!

Answer 9

This is excellent to hear, I enjoyed reading some of the details on how these changes came about. My main question is how does this affect these long-term beta sites? Graduation comes with these changes:

1. New site design
2. Moderator Elections
3. The site is added to the footer list of SE sites
4. They can be a migration target
5. SE removes the Sword of Damocles hanging over the site

For a site that's in this new perpetual beta state, these changes don't happen. Will these sites get some sort of "we're a healthy community, but not graduating" site design to differentiate them from the beta design? Will SE resources do periodic check-ins with site mods? Some of the sites I've been active on have suffered from mods vanishing or just slowing down their activity. On betas this can be a strain for the remaining active mods. Will replacement mods be pro tem, or will elections start happening on these beta sites? Will there be some notification from SE (another kind of periodic check-in, maybe?) that the site is well above the closure threshold?

I suspect that this announcement portends a fresh change in SE's treatment of beta sites, so I'm not expecting an instant answer on this stuff, but it's something that I'd like to know the plan on.

Answer 10

This makes me think the 'Public Beta' phase should be renamed, since it's not really descriptive of the actual status. A beta sounds like something that is unfinished, or a trial. These are 'finished' sites, but with only a handful of people around.

This matters, because if I want to introduce someone to, say, the coffee SE beta, the connotations of 'beta' are that it's unfinished. I'd much rather say it's something else (although I'm not quite sure what).

I appreciate that certain aspects of the community structure are 'unfinished', and there may well be unresolved issues around policy, tagging etc., but hopefully most big ones have been dealt with in the private beta.

Here are some thoughts for better names off the top of my head, I'm sure there are better ones:

• Incubation
• Growth
• Nursery
• Establishment

Answer 11

One metric that I think best represents community health is the number of actively participating users. Users that regularly visit the site to ask questions, answer them, edit posts or perform any other kind of community moderation. Increasing the size of the active community is pretty much the most important aspect of the beta phase.

I suspect that the number of questions correlates with the size of the active community, at least for healthy sites. But I still think that having some measure that represents the number of active users would be far more useful than the statistics we currently have (my old feature request on this issue).

Answer 12

Three out of four recently graduated sites did not match the consistently receive 10 questions per day criteria.

1. When a site starts to consistently receive 10 questions/day, we’ll consider it for graduation.

Namely, Quantitive Finance, Law and Philosophy (365 days interactive version of the chart, DSE query):

Note, how I shamelessly plugged Ethereum beta site statistics into the chart :-)

I don't want to blame anyone and I see that these three communities are very well organized and the sites are healthy and active. But maybe, in 2017, it's time for an even more clear outlook on the health of SE sites ;-)

I suspect size of the community and meta activity is much more of importance than just questions per day. Could we get an update on this? To back up my observations, I'd like to show that Arduino with constantly around 15 QPD but only ~200 avid users is still not graduated. I somehow get the feeling there is a pattern.

If number of active (avid?) users is of importance, I could think of creating fancy scores/ratios for sites which include both QPD and avid users, e.g., SCORE = QPD * AVID. But that's unscientific, let's put this in ratio with days in beta, SCORE = (QPD * AVID) / DAYS. I've picked some of the more active beta sites to pre-compute the score:

1. Ethereum 7.244
2. Sitecore 6.956
3. Politics 4.186
4. Arduino 3.899
5. DataScience 3.273
6. Bitcoin 3.089
7. History 2.925
8. IoT 2.583
9. Monero 2.540
10. Vi 2.524
11. German 2.425
12. CiviCRM 2.365
13. Space 1.773
14. ElementaryOS 1.710

Note, how Ethereum is always on top ;-)

Answer 13

Can I suggest a slight tweak to the "when will a site be closed down" policy? Added text in bold.

If there's enough moderation for a public beta site to consistently remain free of spam, for flags to be cleared, and for our Be Nice policy to be upheld, and if a strong majority of answerable questions are receiving high quality answers, your site will remain open.

I gather that the purpose of closing sites is to protext SE's reputation.

We don't just want a reputation as being free from spam and abuse. We want a reputation as having the best, expert-approved answers. We want people Googling their questions to think "Hey, StackExchange, I'll click that link first!".

I'd argue that if a site is free from spam, clears its flags, deletes or doesn't attract nasty comments etc etc - but never succeeds in actually becoming populated by keen experts in its field and is full of unanswered or poorly answered questions - then that's just as damaging to SE's reputation.

---

I'd suggest basing this partly on questions unanswered % but mostly on a periodic dip-check of content, skewed towards higher-traffic questions. It'll need to be applied intelligently so we don't penalise sites for attracting challenging questions. And there should be plenty of warnings for the community concerned and opportunities / help for them to raise their game.

It's also probably healthy for SE to have people periodically checking this sort of thing even for the quality sites - occasional meta posts like "Hey, this question got 10,000 views but doesn't have any quality answers which risks making the site look bad, can you give it some attention?". Maybe an auto-community-bounty for unanswered/high-traffic... now I'm just thinking aloud.

Answer 14

Using questions/day seems reasonable, but over what timeframe are we calculating the questions/day ratio? This criterion unfairly restricts old beta sites from graduating, because questions/day is essentially a weighted average which is heavily weighted towards the low-traffic beginnings of a site.

Suppose a site is 4 years old and has an average of 1 QPD. That's 1465 Q / 1465 D. Now suppose in its 5th year it gets 10 QPD every day. That's (3650 + 1465)/(1465+365) = only 2.8 QPD! To get up to 10 QPD average, the site would need 46 QPD during that 5th year to get above 10 QPD total! Clearly that is never going to happen for some sites.

I would suggest that if the QPD metric is currently a simple SUM(questions)/SUM(days) it be changed to count QPD over the past 365 days. If a site can maintain 10 QPD for a year, it really doesn't matter what the question rate was in years long gone.

Answer 15

I did some simple exploratory analysis on the proposed criterion (questions per day, QPD). Turns out its logarithm is highly correlated with the logarithm of the user visits per day (VPD) statistic. At 10 questions per day, a site may need to generate about 9000 user visits, which may be beyond the possible audience of some niche sites.

Would recommend:

• quantifying the closing (aka "Broken windows") threshold (as e.g. the backlog of stale moderation/review tasks)
• supplementing the QPD statistic with the number of recently active answering users