Back in April of 2010, Joel shared our assumptions about the role of small sites in the newly minted Stack Exchange network:

If a site does not have enough activity at the end of 90 days, it will be closed down. Any existing Q&A will be archived and made available for download, but the site itself will not remain live. Small, unhealthy sites do nothing but draw traffic away from other sites, splitting audiences, so we don’t want to keep them around.

Past experience had led us to believe that “small” and “unhealthy” would mean the same thing: we took it as a given that quiet sites would develop spam and obvious broken windows, while high quality sites would always grow big and graduate. But you proved us wrong! Five years later, we have lots of tiny sites which have been in public beta for months or years, each consistently producing excellent Q&A which helps people with real problems. Small, consistently active sites are great!

Our old assumptions didn’t pan out, but we never provided clear updated criteria for when we would close or graduate a Q&A site. We're rectifying that now.

The TL;DR:

  1. When a site starts to consistently receive 10 questions/day, we’ll consider it for graduation.
  2. If a public beta site does not produce consistently helpful content, and lacks the caretakers needed for flags and spam to get handled and our Be Nice policy to be upheld, it will be closed.

Is your site in between these two categories? You don’t have anything to worry about. Regardless of how small the site might be, you have a home here in the SE network. Spoiler alert: As of this post, no site which is currently active is at risk of being shut down.

If you’d like to learn more about the nitty gritty details, read on.


Clarifying graduation

While Community Managers have historically had high levels of agreement about when a site should graduate, we arrived at our conclusions via totally different methods. When Pops joined the team after being a longtime user, he noticed this, and decided to examine our assumptions and procedures. He also wanted to apply some lessons from the research showing that well-designed data-backed algorithms and heuristics are more effective at making decisions than human intuition alone.

Graduation is a hard concept to pin down quantitatively, but we basically expect that a graduated site should be able to continue growing and governing itself indefinitely without any outside help unless something truly catastrophic happens.

Pops looked at a wide variety of statistics and how they tracked on sites we've graduated in the past, and discovered that there was a strong correlation between graduation and the number of new questions a site receives per day. Fast forward through all the boring analysis, and here's the result: from now on, when a site starts to consistently receive ten new questions every day, we'll consider it for graduation. This is not 100% automated; CMs will still be manually checking on how individual communities are doing, and some sites might still graduate "earlier" or "later" than their question activity alone would suggest; but it's an effective rule of thumb.

Tangible guidelines for site closure (aka Reasons why you shouldn’t be afraid)

While Pops worked to clarify graduation, I looked at things from the opposite direction: site closure. The precedent was set early on that sites had two options; graduate, or get shut down. This created a lot of unnecessary angst for our smaller and mid-sized communities. Lots of folks on excellent sites assumed that since they hadn’t cleared the graduation hurdle yet, we might come in and turn off the lights at anytime. Not exactly great for morale.

Thanks to many devoted users, it’s grown clear that smaller SE sites can do a great job of maintaining themselves and producing high quality Q&A. Not every site is going to be a blockbuster success, but our small sites are serving their own communities well. We’re proud of you, and we want you here.

What does this mean? If there's enough moderation for a public beta site to consistently remain free of spam, for flags to be cleared, and for our Be Nice policy to be upheld, your site will remain open. However, if community leaders drop off, flags sit without being addressed, and we can’t find any suitable volunteers to step forward, the site gets closed.

As of this post, not a single site currently active in our network is at risk of being closed. Closing public beta sites is a rare occurrence; we expect it to stay that way.

Success and graduation are not the same thing

When the network was just getting started, we all worked off of judgement calls and personal experience because that was all we had. Since then, we've tested our assumptions and we’re excited to have been able to create more objective criteria for some of our most important decisions. The "trial by fire" approach to site launches and graduations was a great early motivator for communities, but in reality, success and graduation aren’t the same thing. Now, communities can focus on Q&A about topics they love, not on whether their site will survive. We’re proud of what you’ve accomplished so far. We hope you are too.

  • 47
    As indicated above, I teamed up with Ana on this, so I'm leaving a comment to be more easily @-pingable. – Pops Jun 1 '15 at 17:35
  • 33
    Skeptics graduated with 5.7 questions per day, and I think there were a few other low-volume sites that graduated. How do you plan on handling those sites that inherently have a much lower question value, and might not reach 10 questions per day any time soon? – Mad Scientist Jun 1 '15 at 17:43
  • 41
    At first glance on your title I expected this post to be about the closing of health.se ... – rene Jun 1 '15 at 17:51
  • 38
    Even the SE team has repeatedly emphasized that the current A51 stats do not matter as to whether a site will be graduated, as there are far more important (and subjective) goals, such as sense of community, self-moderation, user retention, etc. etc. For many sites (such as Programming Puzzles & Code Golf), such an increase in question volume would mean a decrease in overall site quality. – Doorknob Jun 1 '15 at 18:02
  • 23
    Is this 10 questions per day that "survive" -- i.e., aren't closed/negatively scored -- or just a raw 10 posted per day? @Pops – Josh Caswell Jun 1 '15 at 18:22
  • 21
    @Doorknob You're right that graduation needs to happen on a case by case basis. This is why Pops' 10QPD threshold is viewed as the trigger for a conversation of whether a site should graduate, not the decision itself. What about smaller sites which probably won't hit that 10QPD threshold? I say above "Success and graduation are not the same thing". Maybe now we need to look at how non-graduating communities can get recognized for their success. – Ana Jun 1 '15 at 18:47
  • 29
    This feels like a mindset shift to me. Are you saying that it's possible for a site to stay in beta forever, @Ana? – hichris123 Jun 1 '15 at 18:48
  • 44
    @hichris123 "Are you saying that it's possible for a site to stay in beta forever?" Yes. – Ana Jun 1 '15 at 18:55
  • 73
    As a member of the beta site Amateur Radio, I'm very pleased to see this announcement. I've been constantly worried that we'd be shut down because we have little activity, and saw no good way to fix this. With this statement of policy, I am now reassured that I can focus on making our site have high-quality content, not high quantity. – Kevin Reid Jun 1 '15 at 18:56
  • 30
    @hichris123: This might be the moment to say that we have long considered "beta" to be a bit of a misnomer. Many people think of beta a being a very temporary state. I tend to think of beta sites in something of the same category as Gmail, which was in "beta" for 5+ years. That's also part of why the beta theme was redesigned a few years ago. – Jon Ericson Jun 1 '15 at 19:01
  • 22
    @JonEricson Are you (& rest of team) open to suggestions for a better name than "beta"? We could brainstorm something while this post has everyone's attention. – user259867 Jun 1 '15 at 19:08
  • 64
    Too bad that Theoretical Physics SE has become a victim of the earlier bad SE practice of shooting smaller (very high-level expert) communities down because of their naturally longer turning time, disregarding the fact that world class physicists have been active there producing for the international physics community immensely valuable content. I really hope that SE has learned to appreciate smaller high-level expert communities a bit more by now ... – Dilaton Jun 1 '15 at 19:30
  • 80
    Have you thought about whether appointed mods should be "for life"? When beta was only a year or two it didn't matter; does it now? (Real question, not a leading one.) Right now beta mods only leave if they step down or they do something really egregious and get fired, but I've seen some cases where long-term mods were doing harm but not get-fired levels of harm, and the community really can't do anything about that. Nor did it choose them. I don't think you want term limits (let mods who are doing a good job keep doing it), but maybe some way to check in w/the communities periodically? – Monica Cellio Jun 1 '15 at 21:24
  • 15
    Related (maybe we can address this now?): meta.stackexchange.com/q/239464/162102 – Monica Cellio Jun 1 '15 at 21:25
  • 16
    Since no one has done so for four hours now, I've followed Pops's suggestion and made a separate post to discuss what can be done about the beta label. – Martin Ender Jun 1 '15 at 23:57

15 Answers 15

As you seem to be renaming stages anyway, how about uncoupling the different aspects of graduation, i.e., do not enforce them to coincide? As it seems to me right now, there are six effects of graduation:

  • Site Design: This is obviously causing most of the work and it’s reasonable that you only want to do this, if a site has sufficient activity and survival chances to justify the effort. So, this is one thing that should certainly only happen at 10 QPD.

  • Moderator Elections: This does not seem to be causing much work (as it is mostly automatised) and could happen whenever there is need for another moderator, be it because a pro-tempore moderator stepped down or due to increasing site activity – and if certain conditions are met (site age, number of users who can vote), which do not necessarily have to be the graduation criteria.

    It seems that you are open to the idea to hold moderator elections without full graduation anyway ([1], [2]), though on sites which are scheduled for graduation anyway. (I begin to suspect that lacking a moderator becomes a possible way to accelerate graduation, once you are in the queue.)

  • Community Ads: I do not see any reason why those should not be enabled on any site older than a certain age (so that the direction of the site is clear).

  • Reputation Levels: The most important effect of these is that it decides who can review. Thus a large community is required for those levels to be adjusted. Maybe this coincides nicely with graduation, maybe a different criterion is appropriate – I would need more data and experience to decide this.

  • Full migration target: Is this still a policy? I have seen some migrations to beta sites and most of the time the alternative was outright closing the question. I can see that migrating a borderline question to a site that has just reached public beta and thus may vanish soon is a bad idea – but then borderline questions should probably stay where they were intially posted anyway. Anyway, I do not think that graduation is required for this status. Maybe in some cases it need’s reminding that questions can be on-topic on more than one site.

  • Being listed in the footer: I am somewhat indifferent about this.

So for example, how about the following?

  • The actual label beta is applied for about a year, with possible extensions if the site is still struggling, but alive.
  • Afterwards, certain features of graduation (elections, community ads, full migration target, listed in footer) are enabled and the label beta is removed.
  • If the site has reached 10 QPD, the site has won the Internet and gets other features of graduation (individual design, reputation levels).

Note that this is really just an example, I would need more data, experience and insights into SE to tell whether this suggestion is really good.

  • 28
    @HomegrownTomato The footer isn't designed to list every site. Pretty much every category has too many now ("business" and "professional" categories being the exception... in that they are missing from the footer entirely because when it was designed, no graduated sites were in them). We should consider randomizing the list, perhaps, to give all sites a chance... but first, the question is "does anyone even use the footer?". – Adam Lear Jun 1 '15 at 23:08
  • @AnnaLear When it was designed, it did list every graduated site. Now it doesn't... I know I use the footer to go to Area 51, and maybe Stack Apps or Data Explorer. But you have the click data, don't you? – user259867 Jun 1 '15 at 23:11
  • @HomegrownTomato The previous version of the footer listed all graduated sites. Current version does not (by design). I don't think we track clicks from the footer specifically... I'll have to look when I'm at a computer with access to the code. Out of curiosity, why do you use the footer instead of the site switcher for A51/StackApps? – Adam Lear Jun 1 '15 at 23:15
  • @AnnaLear They are conveniently located in the footer on the right ... in the site switcher they would get in the way if pinned, and would take longer to access if not pinned. I don't go there often. – user259867 Jun 1 '15 at 23:18
  • 15
    One other thing you're missing is that sites in beta, like Code Review, are currently somewhat discouraged from being migrated to — even if, like Code Review, they're actually in the queue for graduation. So more clearly distinguishing "beta" from "indefinite okay status" (and being fine with migrating questions to the latter but not so much the former) is another thing to add to this list. – Nathan Tuggy Jun 2 '15 at 2:34
  • @NathanTuggy: Thank you, I was bound to miss something. – Wrzlprmft Jun 2 '15 at 6:53
  • 5
  • As a note, ELU constantly migrates stuff to the Beta site ELL. Linguistics does it to a smaller degree... so it is possible but it requires a diamond to do it. Beta sites don't have any migration targets other than Meta, so you can't migrate away from a Beta site... Diamonds, excepted, I believe in this case, as well... that's actually what prompted the question Josh posted above me here. – Catija Jun 2 '15 at 7:21
  • 12
    Migrations paths from betas/"betas" are another item in the "should be activated"-list. – Raphael Jun 2 '15 at 11:05
  • 1
    Rather than trying to guess the right time to have a sudden increase in rep requirements for the review queues, perhaps the requirement could be based on position rather than rep. So the top 5% of active users have access to the review queue, for example. – trichoplax Jun 2 '15 at 18:20
  • 4
    @Catija: Most sites want interesting questions. Smaller sites especially depend on a small core of active experts to supply good answers. Send in too many questions that don't interest the core expert community, and those experts will get bored and leave, and the site will either decline or, at best, end up radically changing its scope. Alas, preventing that does sometimes mean telling new users "sorry, your question is too low-level / boring / homework-like for this site." At least with ELU/ELL we have another site with a more suitable scope and community to send them to instead. – Ilmari Karonen Jun 2 '15 at 18:55
  • 1
    @AndrewLeach It's pretty common. SO, SU, SF, Sec.SE, U&L, Programmers all have it. Community migration can actually work fairly well in my experience. – cpast Jun 3 '15 at 7:24
  • 3
    I'd like to add one more feature to your list. Today, sites in beta can be shut down at any time, and all their content disappears (well, there's an export, but it won't show up in search). For graduated sites, there's an implicit promise that this won't happen: Stack Exchange will continue hosting the archive of questions & answers. For me, I buy into the mission of SE as building an archive of high-quality content. If there's a significant risk that all my contributions might disappear at any time, that's a deterrent to contributing; knowing that won't happen would be a nice benefit. – D.W. Jun 3 '15 at 20:34
  • 1
    @AchalaDissanayake: The privileges are lost. Privileges are directly linked to reputation, nothing else. – Wrzlprmft Feb 11 at 21:49
  • 1
    @AchalaDissanayake: Right now, it comes with the design. – Wrzlprmft Feb 12 at 8:09

As an active participant in a low-volume, but high-quality (well, mostly!) SE site, this is reasonably welcome news.

One of the most tangible effects of "graduation" is, of course, the bespoke graphics-css-look-and-feel to reflect the ethos of the site's core interest. "Beta" sites (or whatever they might be called in future -- "niche"? "boutique"?) have pleasant but generic and shared graphics-css-etc.

Given the possibility that a site might stay in beta forever, I wonder what the thinking might be about tossing those sites a bone where some css tweak would help that site's Q&As. The case I have in mind for "my" SE site is a small but significant tweak to the font stack, but I'm fairly confident that other beta-forever sites might have a similar need.

Is it too much to hope that the shift in policy announced by Ana♦ (with Pops♦) might also bring some consideration for small but significant CSS tweaks to the beta-forever sites? "Where there's life, there's hope," as Sancho said to Don Quixote...

  • 6
    sucked me back to Hermeneutics...lol. I thought that site was a graduated site. I may have to get more involved there. – Malachi Jun 1 '15 at 20:56
  • 85
    For what it's worth, I've been pushing (what seems like forever) to have the beta label removed when a site is no longer "in beta". We're slowly warming up to the idea... but beyond that, I don't yet know how far I can push this idea that long-standing beta sites are essentially "finished." I envision allowing communities to do minor look-and-feel tweaks: coming up with a logo, a byline under the title, maybe changing a few key graphic elements — it would be more like skinning a site than a complete redesign, but like I said, let's get that meaningless "beta" label removed first! Amen. – Robert Cartaino Jun 1 '15 at 21:31
  • @RobertCartaino - But then what does "beta" mean? I guess that's what you folks need to decide. As far as I can tell, the only obvious difference is that moderators are appointed rather than elected. – T.E.D. Jun 2 '15 at 9:30
  • @T.E.D. - Funny you should ask that. I'm not Robert Cartaino (you might already have noticed), but the related thread is addressing exactly your question. – Dɑvïd Jun 2 '15 at 9:35
  • @David - You are absolutely right. I'd say that entire question is about my question/point. It isn't posed that way (shame), but that's what it boils down to. – T.E.D. Jun 2 '15 at 9:41
  • 13
    @T.E.D. A "beta" site is analogous to the software development model. The beta of a site is when it is still in its formative stages, subject to abrupt changes, and they're still trying to figure out what it is all going to be about. Once a site has settled in to what is pretty much going to be forever, it's time to remove that largely misnomered "beta" label. Typically that happens about 60-90ish days in. That is just my opinion. – Robert Cartaino Jun 2 '15 at 12:01
  • 1
    @RobertCartaino But when do the other "graduated" benefits come in? I'm not sure that 60-90 days is long enough for the graduated rep levels to be reached by a sufficient group... is it long enough for Meta participation to be strong enough for mod elections? I feel that these two, in particular, would suffer under a short beta and that something more like 6-12 months would be preferable. – Catija Jun 2 '15 at 17:49
  • 5
    @Catija No, at about 90 days (TBD) a site would stop being labeled "in beta". Sites get the benefits of graduation when they graduate. Read the original post and then my comment again. There is a large middle class of sites that may never graduate at all. That doesn't mean they are eternally in "beta testing". That's what we have now — to anyone not reading this thread, "beta" sound like these sites are never finished, and that's not true at all. It's the unfortunate use of the label itself that is entirely wrong and misleading. – Robert Cartaino Jun 2 '15 at 18:08
  • 2
    @RobertCartaino I understand that there's a difference between sites that will never graduate and sites that will and I'm ok with the concept of some sites never graduating. I think removing the beta tag would be great. But... is there a reason that these non-"beta", non-graduated sites shouldn't get some of the benefits deemed part of "graduating"? There are, particularly, many sites that are "slated for graduation" that haven't yet due to web-design, so they've been waiting for (in some cases) years in "limbo" because the design process is so slow. – Catija Jun 2 '15 at 18:28
  • 1
    @Catija I'm trying to unbundle many of the features of graduation to allow sites to benefit from the features they need, like elections, but I think a custom design will always be part of that top-tier level (i.e. "graduation"). – Robert Cartaino Jun 2 '15 at 18:41
  • 1
    @RobertCartaino for me anyway, the site design will always be the least important, particularly if it's the only part that takes significant time to implement. – Catija Jun 2 '15 at 18:43
  • @RobertCartaino what about meta.stackexchange.com/questions/211640/… ? – Manishearth Jun 3 '15 at 14:23
  • 2
    @RobertCartaino maybe we could review a beta site after say, a year, and remove the beta label if it does not graduate, while also running an election. – Sklivvz Jun 4 '15 at 23:58
  • @RobertCartaino So, what about sites that have been in beta since 2010 or so? And yes, those do exist. – Powerlord Jun 23 '15 at 18:47
  • 1
    @Powerlord What about them? What we are working on right now amounts to removing the 'beta' label much much sooner and getting rid of the graduation label entirely. Some of the details may change, but those two meta responses pretty much captured the crux of it. – Robert Cartaino Jun 23 '15 at 18:54

On graduated sites the community elects permanent moderators, but on beta sites "pro-tem" moderators are appointed by the SE community managers. This made sense with the original vision of beta -- it'd only be for a year or so, maybe two, and then the community would graduate and choose its own.

With long-running (permanent?) beta sites, you can have moderators who were appointed during the private beta, before the vast majority of the community was present, who remain in the job forever. Sometimes this is just fine -- if a good moderator who remains engaged with the site and serves the community well keeps doing that, we all win. But sometimes, over time, the fit between a pro-tem mod and the site degrades -- a moderator stops coming around as much, lets flags pile up, won't moderate in accordance with community wishes because he personally disagrees with a change in direction, just seems to be a little crispy, whatever. If a moderator behaves in a way that's egregiously bad the community managers might step in, but that's very rare. So things can fester sometimes and that hurts everybody.

One might point out that on graduated sites moderators are also elected "for life" (unless they mess up badly or step down), but there are two important differences. First, on a graduated site there was an established community at the time of the election and that community chose its moderators. That might have been years ago and the community make-up might have changed, but there was some established community making the decision. Second and perhaps more important, beta sites by their nature change over time, and what a community needed on the first day of public beta might not be what it needs three years later. I've been an active participant on several betas, a pro-tem mod on an established one (appointed about two years in), and a pro-tem mod on a brand-new beta. They're different.

With the possibility of appointed moderators serving for many years on evolving sites, I think it serves everybody's interests -- moderators, users, and SE -- to come up with a way to periodically check the community's support of its current moderators -- some sort of vote of confidence for the team. This should allow private response, be solicited (don't just rely on people to click "contact us" if they want to say something), and be done in a way to minimize drama (anonymous survey good, meta voting probably not good). I'm envisioning a poll with two questions: (1) Overall rating for the team (1-5 or whatever), and (2) here, have a textbox.

This isn't an election and the results aren't binding but, as with the site self-evaluations for content, the community managers would use the results of this to help the site in whatever way seems most appropriate -- quietly coach that one mod, add someone new to the team because the workload has gotten high or because a needed skill is lacking, try to contact that absentee mod, tell the mods what they're doing well, nudge someone to start a meta discussion about that theme that came out in a bunch of the comments, etc.

As a user on long-running beta sites I would like us to have the chance to provide this kind of feedback from time to time. As a moderator on such sites I would like to have the chance to receive it.

  • I was going to write something very similar to this, community elected moderators are definitely a way to make everyone feel like they are involved in the site moderation. It helps avoid "moderators vs community" - if that exists, that's a problem that should be addressed somehow. – enderland Jun 2 '15 at 13:50
  • This happened on one beta site recently: 2 of the 3 initial pro-tem moderators stood down (perhaps they had less time to spend moderating, and that was difficult, so they stopped down to let other people do it); the remaining moderator with a community manager asked users to volunteer as moderators and/or nominate other users and/or recommend (by up- and downvoting and by comments) the volunteers and nominees; finally, SE (taking everything into consideration, including all user opinions and maybe the opinions of the past and present moderators) then chose the new pro-tem moderators. – ChrisW Jun 2 '15 at 18:17
  • I don't want to claim that a binding precedent or to make extra work for community managers but maybe it's a good way to go about it: pro-tem users are always allowed to step down and be replaced; it mimics the initial appointment of pro-tem moderators (i.e. people volunteering and SE choosing); it lets everyone (including users, the remaining moderators, and SE) have input; SE keeps the final say in the matter (maybe the community isn't experienced enough or numerous enough yet). – ChrisW Jun 2 '15 at 18:22
  • @enderland Perhaps a way to avoid "moderators vs community" is for moderators to ask their community what policies the community wants, discuss proposals until some consensus emerges, and then obey/implement those community policies. – ChrisW Jun 2 '15 at 19:36
  • 1
    @ChrisW communities should already be discussing their sites' policies on meta. If a site decides that X is off-topic or answers must meet requirement Y etc, then mods receiving flags about X and Y are expected to follow what the community agreed on. Your comments seem to be mainly about how to choose moderators; that's important but not really what I'm talking about in my answer. I'm saying that it would be good for the CMs to "take the community's pulse" from time to time, before you reach a crisis point like two of three mods being completely absent. – Monica Cellio Jun 2 '15 at 19:42
  • +1 thanks for clarifying: you're saying that a current problem on beta sites isn't moderators being appointed instead of elected, rather it's their being permanent instead of renewed; not about how they're replaced if they resign, but about feedback if they're emplaced indefinitely. – ChrisW Jun 2 '15 at 20:33
  • @ChrisW kind of. I'm saying that appointed rather than elected is a problem if it's long-term -- for a year or two doesn't matter but for long-term betas the community should weigh in every now and then, because their needs might have changed or a previously-helpful mod might be having problems that aren't enough to call for a change on meta (awkward, political) but are nonetheless still problems. So SE should ask for feedback now an then, like they do on content (community self-evaluations). – Monica Cellio Jun 2 '15 at 20:43
  • 1
    Hrm I once proposed something similar here, but it wasn't very well received. I like the idea of doing a yearly Yes/No (or Up/Down) vote for renewal of moderators, and if someone has their No's significantly outweight their Yes', then maybe it's time to reevaluate if that moderator should step down so someone else from the community can take over. – Rachel Jun 3 '15 at 14:56
  • 2
    High level, theoretically, it's a great idea. Ironically, personally for my experience on score of Beta and nonBeta sites, I usually find permanently appointed proTempore moderators to be between good and unobjectionable (with only 1 exception); but saw plenty of bad apples among elected ones. – DVK Jun 8 '15 at 3:03
  • @MonicaCellio - this (moderators following the community) is a desired model. Sadly, I saw it sometimes violated (with zero reaction from SE team) in both ways - either moderators ignoring the community; or a very persuasive and eloquent moderator taking it upon themselves to push for a particular course, and not letting a community the chance to form their own opinion (not many people are open to arguing against an eloquent position stated by a forceful moderator; and even fewer are good enough to succeed even if they are open given moderators built in popularity advantage). – DVK Jun 8 '15 at 3:08

This is awesome. Thank you.

For quite some time, I've been worried about how a couple of the beta sites I'm active in are doing. This makes me feel like they're just fine, given that they match the criteria you gave. Thank you guys for the overhaul.

However, can the word be spread about this? This question has been getting a lot of attention, and it is, after all, , but there are many people who don't check out Meta Stack Exchange (sadly).

My proposal for that is that the Area 51 stats be updated a bit. They can all remain - I find that they're quite nice - but keeping the text as is can make people worry. Seeing the "needs work" on the Views category isn't encouraging - but now that we know that that isn't necessarily a problem, it would be nice if that knowledge was emphasized.

Seeing things in the red is a bit scary. Emphasizing that lower stats are just fine can fix that.

Imported from my comment:

For the "% answered", "users", "answer ratio", and "visits" - and perhaps "questions per day" - I would suggest getting rid of the "needs work" label, and perhaps the "okay" one. At the very least, changing the colors from red would be nice, and a link to this question would be helpful.

  • 16
    I definitely agree. For smaller sites, the A51 stats page can be... alarming. – Zyerah Jun 1 '15 at 20:48
  • 3
    It does show up in the featured box of each individual site: i.stack.imgur.com/frTeD.png – user160606 Jun 1 '15 at 21:10
  • 1
    @Christofian I did forget about that, thank you. I was taking the long term into account, though - this post won't stay featured forever. – HDE 226868 Jun 1 '15 at 21:10
  • 9
    I hear you. The way A51 stats get displayed currently are really set up for the early 'trial by fire' mentality. What specifically would you change? – Ana Jun 1 '15 at 21:15
  • 1
    @Ana For the "% answered", "users", "answer ratio", and "visits" - and perhaps "questions per day" - I would suggest getting rid of the "needs work" label, and perhaps the "okay" one. At the very least, changing the colors from red would be nice, and a link to this question would be helpful. – HDE 226868 Jun 1 '15 at 21:17
  • 17
    @Ana In my opinion, it's less the thresholds themselves, and more the scary text that accompanies the thresholds. If that text could convey somehow that the site won't shut down and the site is "on-track", I think the starts would be more useful and start less unproductive meta conversations about how to arbitrarily raise the stats. – user160606 Jun 1 '15 at 21:19
  • 1
    Related (on Area 51): meta.stackexchange.com/q/239464/162102 – Monica Cellio Jun 1 '15 at 21:27

This is a step in the right direction

Having just participated in a beta site myself, the process isn't as user-friendly as it should be. I've lost track of how many times people have unrealistic expectations about how the site will grow. I have a few suggestions on how Stack Exchange can do that:

  1. Could you change the 7 essential questions link that features on every beta site's meta? It creates unrealistic expectations for new users about how much activity the site will have.

    • Similarly, could you add a disclaimer to old blog posts discussing site closures so it's clear that closing sites solely because of low activity is no longer an SE policy.
  2. Could this graph be linked to prominently in new private betas? I think it would go a long way in clearing up unreasonable expectations about beta site activity.

  3. Provide more resources to help communities promote their sites. I have no idea what this would look like (maybe just a simple guide on how to effectively use social media and how to invite new users to join the site). If community members have "easy" actionable ways to increase their site's activity, then maybe people will stop asking questions about the site's activity and start doing something about it.

  4. As other people have already said, the Area51 stats page should be more user friendly.
  5. While I'm at it, a next generation Area 51 would be nice too (though not nearly as important as the first four suggestions).

Thank you for doing something about this, and for taking the time to put many people's minds at rest. Everyone who participates in small beta sites appreciates this.

  • ...so it's clear that closing sites is not longer an SE policy. - I don't think that is an accurate statement. It isn't that they won't be closing Beta sites, it's that none are on the chopping block right now. If a current Beta dies on the vine, I'm sure SE reserves the right to prune the dead fruit. – Pᴀᴜʟsᴛᴇʀ2 Jun 5 '15 at 10:10
  • @Paulster2 sorry, edited. What I meant was the SE should indicate that they wont close sites solely because of site activity. – user160606 Jun 5 '15 at 20:58
  • No worries ... not a big issue. – Pᴀᴜʟsᴛᴇʀ2 Jun 6 '15 at 1:45

Awesome to hear.

With the new criteria for site closure, is there a chance the previously closed-without-meaningful-warning Answers.OnStartups Stack Exchange could be re-opened?

I know there's a new "Startups" beta that's been opened; it's doing significantly worse compared to the original.

New Startups SE stats vs old

I suspect it's at least partially because so many of us from the original SE site don't care to participate after all our work on the original was unceremoniously trashed.

Giving us our profiles and our content back, merging it into the new site, could bring back a sizable community to what was a valuable site in the SE network.

  • 4
    We want to give existing communities every possible chance to succeed and be awesome, but we won't be reopening closed sites. Sorry to disappoint. – Ana Jun 2 '15 at 16:13
  • 11
    749 avid users (read: over 200 rep) out of 18k+ isn't exactly "active" and "doing well". The A51 metrics are spectacularly ill-suited for giving an accurate picture of a site's overall health. Answers.OnStartups wasn't retaining new users, and the community has practically disintegrated. More details here: discuss.area51.stackexchange.com/a/12630. I really hope the second attempt will do better. That's not unheard of. Not every community gels on first try. – Adam Lear Jun 2 '15 at 17:49
  • 2
    I don't agree that the active user to signed-up-at-one-point user ratio is a good metric either though. A community's strength should be measured based on the number of users it actually has, "hit and runs" aside. By no means would I call 750 a massively thriving community, but it beats the pants off of 110. Even percentage wise, the old site has the new one beat. Were you there at the end of Answers.OnStartups? I don't remember you. What I remember was that in the last month or so, we had picked up a surprising number of new enthusiastic contributors. There was no disintegration. – rbwhitaker Jun 3 '15 at 3:08
  • 5
    @rbwhitaker I assume your comment is aimed at me. I never participated; startups aren't my thing. But, I did watch - that was my job as a community manager. I watched moderators go inactive, I watched issues not get handled except by one overworked person for a long time, I watched us be unable to find new moderators to supplement the team. Having said that, we did learn something from Startups as well as other sites we shut down or let pass in the past. I'm not saying we were always right, but none of those decisions were made lightly and those lessons are where these new guidelines are from. – Adam Lear Jun 3 '15 at 5:25
  • 1
    @ Ana @Anna Lear - I understand these are difficult decisions. I still wish the communication had been handled better, and the community had been given a chance to fix the issues before our content was trashed. But focusing on the present and future, there are three things that suck about this situation that I hope you'll fix: 1) If the Area 51 metrics are so shitty, why are they still the only ones being displayed? 2) In addition to metrics, why is there a such a gap in standards (749 avid users = "excellent" according to Area 51, being derided here by Anna)? (continued..) – Jay Neely Jun 4 '15 at 20:49
  • 1
    3) "Not a single site currently active in our network is at risk of being closed," so the current Startups SE presumably has its moderation in good order, even if by all the public metrics (which surely are good for something?) it's doing poorly in other areas. I understand you don't want to re-open closed sites. But if you want the existing one to succeed, are you really dismissing out of hand, without discussion, the idea that bringing at least profiles / user accounts from the old site might attract a bunch of us back to the new one, and give it a valuable boost? – Jay Neely Jun 4 '15 at 20:49
  • 1
    @JayNeely Re: "why are metrics still being displayed?" - Area 51 is pretty much in maintenance mode and we avoid making changes to it unless things are dire. Having said that, I agree with you. We need to do something on the proposal page that sucks less and I can think of a couple things that we could do that wouldn't be very complex in terms of work required. – Adam Lear Jun 4 '15 at 21:02
  • 5
    @AnnaLear - tangential question: were you open about the issues with that site's moderation situation? I never ventured onto that specific site; but I can tell you that there are less-active sites which I like where I (a) would not even remotely volunteer to moderate in normal case - I even rejected an SE offer to be ProTempore once [as I'm by nature unsuited to moderator position] BUT (b) would volunteer in a flash if I was informed that what stands between the site I care about being closed vs. open is availability of such a volunteer and nobody more qualified is available. – DVK Jun 8 '15 at 3:15
  • 1
    Jay and @rbwhitaker - I'm not involved in either Startup instance, but that's very sad. Have you made a request of SE to "Giv[e] us our profiles and our content back, merging it into the new site...."? I can't find any meta posts on the subject, but I would hope (and vote) that such a thing could be done for any closed sites that are resurrected. – feetwet Sep 11 '15 at 14:21
  • Well I think it deserves a straight answer, so here's a straight question! – feetwet Sep 11 '15 at 20:45
  • ... and, as you predicted, here's Robert Cartaino's unsatisfying and tangential response. – feetwet Sep 12 '15 at 0:05

For sites staying in beta, please consider proposals for Allowing CSS customization of beta sites.

I think this is a good start. More clarity = happier users, nearly universally.

However, I think that more feedback would be very helpful, particularly on sites that have been around for years and are still in beta. Parenting is my example of this, but I'm sure there are other similar sites: sites that are in beta for many years, and have reasonably active communities, but have one stat they just don't ever seem to get high enough - in Parenting's case, specifically, questions per day.

It would be helpful to have specific feedback at the per-site meta on a regular basis - quarterly, I'd say, if possible - as to how the site's doing and what the long-term plans are. Right now it sounds like permanent beta is where we'll be at Parenting, from the above, and that's probably fine; it sounds like the issue SE has with graduating has to do with picking which sites to spend the (expensive) resources on to give flashy new designs, and I'm at least not particularly concerned with that side of things. But I don't know if that's true, in our site's case. Maybe there is a plan to graduate us at some point. Maybe they'd graduate us if we did X, Y, or Z, that don't specifically involve increasing Q/day. That not knowing can be frustrating at times.

So - thanks for the above details, very much appreciated - but maybe somebody would stop by every so often and talk to us? Thanks!

  • I will say that I'm not a mod, and I don't know if the mods have this kind of feedback and just don't share it; that would surprise me, given our mods are pretty communicative, but they're also relatively new so perhaps this is something that happens over time. So please let me know if I'm just off base here. – Joe Jun 1 '15 at 21:18
  • I don't think that mods know about how well a site is doing relative to the optimal place for graduating, but it could just be that nobody tells me anything. :-) On a serious note, excellent idea. – HDE 226868 Jun 1 '15 at 21:21
  • 4
    Would even just hearing 'No changes, but you're still doing great.' be more helpful than hearing nothing at all? – Ana Jun 2 '15 at 23:55
  • 1
    @Ana A bit, but the issue for me at least is knowing what path we're on. For a site that's been around a long time, in particular, are we basically expected to be permanent beta? Are we likely to graduate if some number changes a bit? We are not likely to hit ten q/d ever in my view, so while this post was helpful to know we aren't going away necessarily, not knowing if we ever plan to graduate is the real problem to me (and not knowing if there's anywhere we should specifically improve). – Joe Jun 3 '15 at 0:18

This is excellent to hear, I enjoyed reading some of the details on how these changes came about. My main question is how does this affect these long-term beta sites? Graduation comes with these changes:

  1. New site design
  2. Moderator Elections
  3. The site is added to the footer list of SE sites
  4. They can be a migration target
  5. SE removes the Sword of Damocles hanging over the site

For a site that's in this new perpetual beta state, these changes don't happen. Will these sites get some sort of "we're a healthy community, but not graduating" site design to differentiate them from the beta design? Will SE resources do periodic check-ins with site mods? Some of the sites I've been active on have suffered from mods vanishing or just slowing down their activity. On betas this can be a strain for the remaining active mods. Will replacement mods be pro tem, or will elections start happening on these beta sites? Will there be some notification from SE (another kind of periodic check-in, maybe?) that the site is well above the closure threshold?

I suspect that this announcement portends a fresh change in SE's treatment of beta sites, so I'm not expecting an instant answer on this stuff, but it's something that I'd like to know the plan on.

This makes me think the 'Public Beta' phase should be renamed, since it's not really descriptive of the actual status. A beta sounds like something that is unfinished, or a trial. These are 'finished' sites, but with only a handful of people around.

This matters, because if I want to introduce someone to, say, the coffee SE beta, the connotations of 'beta' are that it's unfinished. I'd much rather say it's something else (although I'm not quite sure what).

I appreciate that certain aspects of the community structure are 'unfinished', and there may well be unresolved issues around policy, tagging etc., but hopefully most big ones have been dealt with in the private beta.

Here are some thoughts for better names off the top of my head, I'm sure there are better ones:

  • Incubation
  • Growth
  • Nursery
  • Establishment
  • 4
    meta.stackexchange.com/questions/257652/… – user160606 Jun 2 '15 at 14:45
  • 2
    I've been on several sites on day 1 of public beta and, trust me, they're still very much in flux then. Coffee's been out for a few months so has had time to settle down, but since you won't want a long private beta, there's still a time when "beta" is a meaningful description. Just not for years and years, on most sites. – Monica Cellio Jun 2 '15 at 15:21
  • @MonicaCellio I agree, so maybe there's an extra step? A sort of kindergarten graduation? – fredley Jun 2 '15 at 15:48
  • @fredley I love your idea of multiple stages; however, could you maybe elaborate (even a sentence) as to what each stage would mean? Like, does establishment differ from graduation? etc etc – onebree Jun 3 '15 at 13:28
  • @onebree So the way I'd see it is that after 90 days (or so, depending on the state of the site/community), the beta label is dropped. From an A51 perspective this new phase between public beta and graduation can have a different name, but it wouldn't necessarily be prominent, just a term used to describe the phase. Does that make sense? – fredley Jun 3 '15 at 13:53

One metric that I think best represents community health is the number of actively participating users. Users that regularly visit the site to ask questions, answer them, edit posts or perform any other kind of community moderation. Increasing the size of the active community is pretty much the most important aspect of the beta phase.

I suspect that the number of questions correlates with the size of the active community, at least for healthy sites. But I still think that having some measure that represents the number of active users would be far more useful than the statistics we currently have (my old feature request on this issue).

  • 3
    This propesed measure of health is still purely based on quantity and completely neglects the importance of quality. There does not have to be anything wrong with small (high-level expert) communities by definition (such as TP.SE has been!). A small site whose activity is more appropriately described by days per question than questions per day is perfectly healthy too, if the content is high-level, most of the questions can successfully get answered by experts after a reasonable time, and the community living on the site is happy and satisfied. – Dilaton Jun 2 '15 at 11:41
  • 4
    @Dilaton The important measure is growth or decline of active users, not the absolute level. If users keep coming back to the site, it probably has some use to them. If the number of active users keeps declining, the site has a problem. – Mad Scientist Jun 2 '15 at 12:05
  • 1
    I assume the criterion is 10 questions/day AND sufficiently many users to answer those questions AND enough "avid" (reputable) users to form a core of users who will still be able to pseudo-moderate (e.g. vote to close etc.) after the reputation thresholds increase (which the thresholds do, quite a lot, when the site graduates). – ChrisW Jun 2 '15 at 12:13
  • 1
    @MadScientist I agree that a decline of active users is not so good, but in particular communities that exist since a longer time may also reach a more or less stable equilibrium, as it is might be the case for example with MathOverflow. Such communities can be healthy and successful with respect to what they want to achieve too, even though the net number of active users is not growing much. – Dilaton Jun 2 '15 at 12:16

Three out of four recently graduated sites did not match the consistently receive 10 questions per day criteria.

  1. When a site starts to consistently receive 10 questions/day, we’ll consider it for graduation.

Namely, Quantitive Finance, Law and Philosophy (365 days interactive version of the chart, DSE query):

365 days questions per day

Note, how I shamelessly plugged Ethereum beta site statistics into the chart :-)

I don't want to blame anyone and I see that these three communities are very well organized and the sites are healthy and active. But maybe, in 2017, it's time for an even more clear outlook on the health of SE sites ;-)

I suspect size of the community and meta activity is much more of importance than just questions per day. Could we get an update on this? To back up my observations, I'd like to show that Arduino with constantly around 15 QPD but only ~200 avid users is still not graduated. I somehow get the feeling there is a pattern.

If number of active (avid?) users is of importance, I could think of creating fancy scores/ratios for sites which include both QPD and avid users, e.g., SCORE = QPD * AVID. But that's unscientific, let's put this in ratio with days in beta, SCORE = (QPD * AVID) / DAYS. I've picked some of the more active beta sites to pre-compute the score:

  1. Ethereum 7.244
  2. Sitecore 6.956
  3. Politics 4.186
  4. Arduino 3.899
  5. DataScience 3.273
  6. Bitcoin 3.089
  7. History 2.925
  8. IoT 2.583
  9. Monero 2.540
  10. Vi 2.524
  11. German 2.425
  12. CiviCRM 2.365
  13. Space 1.773
  14. ElementaryOS 1.710

Note, how Ethereum is always on top ;-)

Can I suggest a slight tweak to the "when will a site be closed down" policy? Added text in bold.

If there's enough moderation for a public beta site to consistently remain free of spam, for flags to be cleared, and for our Be Nice policy to be upheld, and if a strong majority of answerable questions are receiving high quality answers, your site will remain open.

I gather that the purpose of closing sites is to protext SE's reputation.

We don't just want a reputation as being free from spam and abuse. We want a reputation as having the best, expert-approved answers. We want people Googling their questions to think "Hey, StackExchange, I'll click that link first!".

I'd argue that if a site is free from spam, clears its flags, deletes or doesn't attract nasty comments etc etc - but never succeeds in actually becoming populated by keen experts in its field and is full of unanswered or poorly answered questions - then that's just as damaging to SE's reputation.


I'd suggest basing this partly on questions unanswered % but mostly on a periodic dip-check of content, skewed towards higher-traffic questions. It'll need to be applied intelligently so we don't penalise sites for attracting challenging questions. And there should be plenty of warnings for the community concerned and opportunities / help for them to raise their game.

It's also probably healthy for SE to have people periodically checking this sort of thing even for the quality sites - occasional meta posts like "Hey, this question got 10,000 views but doesn't have any quality answers which risks making the site look bad, can you give it some attention?". Maybe an auto-community-bounty for unanswered/high-traffic... now I'm just thinking aloud.

  • I'm going to have to think on this further. If you notice in the TL;DR up top, I say the following about site closure "If a public beta site does not produce consistently helpful content, ..." That was a nod toward the notion you raise here. Essentially, we want our Q&A sites to help real people with real problems. What are the benefits to defining this a tad more rigidly, per your suggestion? – Ana Jun 2 '15 at 23:42
  • I don't really know how policies like this are implemented or formalised on your side, I just wanted to be sure that decisions will be content led :-) we sometimes have a tendency at SE to focus so much on moderation, we forget that we're here to give good answers to questions! – user568458 Jun 3 '15 at 9:41
  • 2
    I disagree. On Software Recommendations.SE we have a low answer ratio but that lies in the nature of the topic. (As one user put it: There are infinite requirements, but finite developers) So, having this as a strict rule will shut down useful, active and healthy sites. – Angelo Fuchs Jun 3 '15 at 15:00
  • protext -> protect – Faheem Mitha Aug 18 '15 at 20:49

I did some simple exploratory analysis on the proposed criterion (questions per day, QPD). Turns out its logarithm is highly correlated with the logarithm of the user visits per day (VPD) statistic. At 10 questions per day, a site may need to generate about 9000 user visits, which may be beyond the possible audience of some niche sites.

Would recommend:

  • quantifying the closing (aka "Broken windows") threshold (as e.g. the backlog of stale moderation/review tasks)
  • supplementing the QPD statistic with the number of recently active answering users
  • 2
    Since you mention the log QPD - log VPD relation, here is the corresponding plot that I also mentioned under the question. (It auto-updates.) – user259867 Jun 2 '15 at 22:05
  • 1
    I think you have the cause and effect reversed a bit. Most questions on most sites are viewed by non-users as search results. So the more questions a site has, the more likely they will show up in search results. If the audience is small, but the site is active, they might hit 10 questions/day without needing many visits. Secondly, there's no particular need for sites to graduate. If it's not got serious problems like spam proliferation, it's not going to get shut down. That said, I do think we will discover other criteria besides the rate of questions as we go. – Jon Ericson Jun 2 '15 at 23:22
  • 2
    @JonEricson - made no inference on causation; have not quantified size of the expert core for each site; core experts do view graduation as a great event; thanks a bunch for consideration. – Deer Hunter Jun 2 '15 at 23:27
  • @HomegrownTomato Your plot is great, but I need to zoom out a lot to view it on my whole screen. Is it possible that you make the dots smaller, and show the site on hover? (I tried viewing it on a 20-22 inch screen) – onebree Jun 3 '15 at 13:00
  • @onebree In Chrome, you can zoom out pressing Ctrl and -. Personally, I just scroll around, focusing on this or another area of the chart. The issue with making plot smaller is that many sites occupy nearly the same position; regardless of the size of dots, they would be impossible to resolve. – user259867 Jun 3 '15 at 13:04
  • @HomegrownTomato I can zoom out in Firefox fine, it's just hard to get an overall impression by scrolling. (For me at least). Great job with the plot though. How did you make it in Google? – onebree Jun 3 '15 at 13:15
  • @onebree It's ImportXML and some other Google Spreadsheets machinery; I described the process in a blog post. – user259867 Jun 3 '15 at 13:19

Using questions/day seems reasonable, but over what timeframe are we calculating the questions/day ratio? This criterion unfairly restricts old beta sites from graduating, because questions/day is essentially a weighted average which is heavily weighted towards the low-traffic beginnings of a site.

Suppose a site is 4 years old and has an average of 1 QPD. That's 1465 Q / 1465 D. Now suppose in its 5th year it gets 10 QPD every day. That's (3650 + 1465)/(1465+365) = only 2.8 QPD! To get up to 10 QPD average, the site would need 46 QPD during that 5th year to get above 10 QPD total! Clearly that is never going to happen for some sites.

I would suggest that if the QPD metric is currently a simple SUM(questions)/SUM(days) it be changed to count QPD over the past 365 days. If a site can maintain 10 QPD for a year, it really doesn't matter what the question rate was in years long gone.

  • 11
    It's a moving average. How many questions over the last week or month or quarter. It's not simply (# of questions)/(age of site). That's why they say 'consistently reaches 10 questions per day'. – Keen Jun 2 '15 at 2:50
  • 1
    @Keen in that case, that statistic needs to be shown somewhere. Area51 only shows the simple (#Q)/(days) number. – nhinkle Jun 2 '15 at 3:02
  • 4
    I created a SEDE query for this. – nhinkle Jun 2 '15 at 3:10
  • 2
    @nhinkle: Setting the query for the full 1224 days shows 1.54 QPD, higher than the displayed QPD of 1.2 and almost as high as the default 365-day QPD of 1.65. (Reducing it to 90 days shows 1.34, which is disturbing, and at 45 days it matches the display with 1.2. So perhaps that's already implemented with a rolling 45-day window, not a 365-day window.) – Nathan Tuggy Jun 2 '15 at 3:20
  • 8
    @NathanTuggy: I believe the Area 51 average is calculated based on the last 7 days. It certainly changes from one day to the next more dramatically than would be possible for an entire site lifetime of data. (Note: the public data is always a few days old, so queries against it won't be exact.) The criteria we use is based on a sliding window over several months to avoid spikes such as when students return to school and that sort of thing. Pops did quite a bit of backtesting on the criteria. – Jon Ericson Jun 2 '15 at 5:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .