What should the criteria be for Stack Exchange sites to leave beta?

Question

Today SE removed the "beta" label from 29 established sites, recognizing a need that has been plaguing many of our sites for years. Until today, under SE's rules there were two states: "graduated" if a site meets the requirements and "beta" otherwise. One of those requirements is to sustain an average of ten questions per day, which just doesn't work for some of our smaller sites. High-quality sites have thus been branded with the "beta" label for, in some cases, almost a decade, long after there were questions about those sites' quality, scope, and ability to maintain their communities.

All that changed today -- yay!

Well, it started to change. In the announcement, Catija says that this is a start and SE is still working out what the rules and thresholds ought to be. Basically, we don't know what the criteria are, but by any reasonable criteria these 29 sites qualify, so let's start with these ones. We've been wanting to break up with graduation for a while, moving from this binary system to one where milestones happen when they make sense, but nailing down the specifics is hard.

I've been one of the most vocal people about how the "beta" system is broken, so the least I can do is try to help fix it. In this question, let's focus on what the criteria should be for removing the "beta" label from a site. (We can discuss other milestones, like when to raise privilege levels or when a site gets community ads, separately.)

When a site is brand new it's clearly a beta. The community has to work out through actual use what the scope is, what requirements to apply to questions or answers, and so on. It has to build a community of people working together, on meta and on the main site, to classify, improve, and curate content. It has to figure out what it wants to be. As a community grows, as the body of Q&A grows, and as time passes, what the site is starts to firm up. Eventually you have very-old sites clamoring to advance. It would be good to have some heuristics to indicate both to a community and to SE that a site is approaching, or has reached, the point where the "beta" label should come off.

What are those heuristics? When is a site no longer beta?

In a post on Writing Meta a year and a half ago I outlined some things that I felt justified our advancement. My list was subjective (there's nary a number in the whole lot!), but here are some factors to consider:

• a sizable body of questions, with old ones still getting new answers, and a steady flow of new questions (how do we recognize this? what's "sizable"?)
• content quality (measured how?)
• engaged users who not only ask and answer but also guide new users, review, curate, and use their moderation privileges
• an active meta
• enough users at various reputation levels to support higher privilege levels
• enough active users to support moderator elections
• site longevity: seven years is too long, seven weeks is too short -- when has a site stood the test of time?

What factors should communities and SE be looking at? How do we know when a site is "baked"?

Answer 1

Avid Users

We need a better idea of how many avid users a site has.

Right now Bob who answered one question 5 years ago that hit the Hot Questions list and got 200 reputation from it before disappearing off the face of the earth is counted as an avid user.

Users who show up and write one awesome answer and then leave are great but they really aren't involved in the day to day moderation such as editing/voting/closing etc.

I think that a better definition of Avid Users would include,

• Interacting with the site in some way commenting/asking/answering/voting/flagging/reviewing etc.

• Within a recent amount of time say last 30 days for example.

Right now the number of Avid Users only goes up with time, this way it could go down as well if users stop showing up and participating. Letting it fluctuate with the number of people who are participating in a recent amount of time would give us all a much better idea of how the site is doing.

Answer 2

This question is basically asking something much more abstract: what's the purpose of beta? That is, why do we need this beta state to live between Area 51 idea and a "non-beta" site?

This question matters even more now, since the "beta" designation is now decoupled from most of the things we presently recognize as being consequences of being in beta: appointed moderators, lower rep thresholds, etc.

As I understood it, the thinking was that beta was about doing the following:

• Firmly establishing the scope and domain of the site. What is on/off topic, what do good/poor questions look like, how much subjectivity do we allow, etc. This should ultimately result in a relatively clear set of enforceable rules, typically on the site's Meta.

• Proving that the community is functional and growing. This includes having enough questions being asked that new answerers have a place, a body of experts that has room to grow, maintenance tasks being performed accurately, etc.

The recent release of beta sites seems to be designating "beta" is being primarily or solely about the first one. Most of those sites have clearly figured out what they're about, but many of them function at a lower level in various ways than most previous sites functioned at upon graduation.

So, we could essentially decide that beta is mainly about scope, with the matter of the functioning of the site having a much lower bar.

Scope definition is easy to determine. The site needs to have had enough questions, both open and closed, to allow for a large enough sample size to know what the site is about. There must be meta discussions about scope, with sufficient participation to have some idea about what the general consensus is.

That brings us to the question of how we define "functional". There are a number of criteria which could be objectively measured:

1. User growth is an important aspect to any site. A site which isn't growing its base of active users is set for slow decline.

   So we should determine the number of users per-month who transition into being active users of the site, relative to the number of total active users. "Active user" of course is something that would have to be nailed down, but visiting the site regularly (every few days) and making some number of posts within that monthly timespan would be a good start.

2. One characteristic of smaller sites (or just overly narrow topics) is that there are a small cabal of answers who are involved in almost every question. This would be fine for a single tag, but on a whole site, it is indicative of an insular, isolated community of a small number of experts feeding on a drip-feed of questions. It also makes it difficult for new experts to appear, since if all the questions are handled by the existing experts, there's nothing left to do.

   To detect this, we could find take the reputation of the answerer for each answer, then compute both the average for a week and the median. Note that this counts on the basis of the answer, so one high rep users answering 3 questions would count their rep 3 times. The lower the median is to the average, the less likely it is that a small cabal of users are dominating the site.

3. Community question closure. Elected moderators are exception handlers; in a well-functioning site, questions get closed reasonably quickly by regular users performing close votes. If moderators have to spend time patrolling the site just to make sure questions are being closed expeditiously, that's a sign that the site lacks sufficient engaged users willing to perform "curation" tasks.

   One could measure the ratio of questions closed by moderators vs. those closed by the community. The focus of the measurement should be posts that only get 1 or 2 non-moderator close votes, relative to the number of closed questions.

4. Community consensus on closure. A well-functioning site is one where the community is pretty clear on what is an appropriate question and what is not, and they will vote accordingly. So finding a way to measure this consensus would be useful.

   One objective measurement might be looking at how many close votes age away over a particular span of time, relative to the number of closed questions. Of course, the number of votes should also be taken into account; a spurious single close vote shouldn't count as much as 3 or 4 close votes.

Answer 3

A while ago graduation was based on a fairly thorough manual review. Back then I once checked a bunch of graduated sites, compared them to older sites that appeared to be stuck in beta and tried to figure some patterns that could make reviewers feel that particular site is good enough to graduate.

The only thing I saw in common in these manually approved sites was that all of them seemed to have "question bank" above 3K (total questions asked excluding any that have been deleted). Upon thinking about this a bit I decided that this could make a fairly solid indication that topic covered by the site is good enough to provide an extensive, reliable knowledge base, meaning it's reasonably safe to graduate.

I would prefer that our criteria for leaving beta would somehow take this metric into account, because it appears to be sort of "verified" by past manual reviews.

---

In case if we would want to get sites out of beta automatically, I would probably prefer a bit stricter criteria that additionally accounts for auto-delete features of the system, something like requiring above 3K questions that are older than, say, one year (better one and half, to let 365-days cleanup scripts make reasonable impact).

In this case another possible additional criteria could be some minimal activity check - if we would want to prevent automatically promoting sites that have lots of old questions but weren't active for weeks or months.

Answer 4

TLDR:

---

Looking at the objective of Stack Exchange, we are different from other forums and QA sites in the case that a community is focused on particular topic and provides quality and reliable information about that topic.

Stack Exchange is a network of 174 communities that are created and run by experts and enthusiasts like you who are passionate about a specific topic. We build libraries of high-quality questions and answers, focused on each community's area of expertise.

This indicates that the essential things for a beta site in order to be recognized as full member of Stack Exchange Network: well defined scope and have considerable quality question answer. If a beta site (no matter how many years in beta) has been suffering for these two issues, then it doesn't meet the graduation qualification. Another unique feature of Stack Exchange is "Moderation". If on-topic is a bhakti, high-quality is jñāna​, moderation is yoga​ and all three are required to attain the goal.

We have The Real Essential Questions of Every Beta which applies as to do list as soon as proposal enters in beta and the same can be applicable as check list for reviewing the health of beta. Including the points which are discussed, following is criteria according to my understanding of Stack Exchange.

1. Well defined scope

This is 1st issue listed for private beta. As public beta site grows, more user joins the site and site usually faces the issue of on/off topicness for various subjects. So, it's desired that several major issues should be discussed on child metas and most of issues should have solved with enough consensus on child meta e.g this or this. Consequently, the possible issue should be addressed and clarified with respective meta links e.g this on /help/on-topic page of help-center.

2. Well organized subjects

Tags are subjects of site. So, a beta site should have well defined tags with tag synonyms. Many a times, it is observed that users don't take the tagging issue seriously. So, the issue of improper tagging, removing unnecessary tags and that for tag synonyms should be contained by child meta. e.g. this, this Another thing is considerable subjects should have been discussed on main site e.g if a site is about religion then philosophy, scriptures, sects, history, mythology, deities all the various topics should have discussed. In other words, a site should contain all types of question answers. This will make a site complete in its subjects. 100 distinct tags with well described excerpt, all having with at least 20 questions tagged.

3. Documentation

Apart from generic help-center, a beta site should have guidelines for new users for asking and answering questions in accordance with the objective, perspective and usefulness of community. e.g. this. The child meta should contain FAQ referring scope, guidelines and established policies. e.g. this

4. Number of Users

This is the soul of community without which it can't live effectively. The effective numbers are Avid users stated on Area51. Here I suggest to have good number (say 70+) of "Established Users" (who is having 750+ reputations). Analysis of number of active users are available for moderators site analytics. 30 users active for 5 days in a week would be a mark. Another thing is how many users are actively participating on moderation activities i.e reviewing, flagging etc. Check out how many times a "reviewer badge" is awarded. I think number 20+ would be a good sign.

5. Visits and Votes

As stated by Area51, "A great site benefits people outside the community. Eventually, 90% of a site's traffic should come from search engines" I think 5000+ visits per day is a good sign. 5 questions with 50+ upvotes would be a good sign.

---

^{Note that I have linked some meta posts from the site I moderate as an example only.}

Answer 5

Review Time

How long on average does it take for a post to go through the review queues?

• The faster rude/spam/low quality posts get deleted the better.
• Closing posts before they get answers helps everyone while reopening them quickly encourages fixing them.
• Quick turnaround on suggested edits encourages the editors.
• Helping new users by reviewing and editing their posts helps them get started.

On the other hand if posts are taking forever to get reviewed or it has to wait for mod to review them is much worse off.

Getting things reviewed in a timely matter (I would argue no more than 48 hours) is a sign of a healthy community especially if the community is able to handle the reviewing without moderator intervention.

Of course, all that requires active users with enough reputation to handle the review tasks which is also a sign of a healthy community.

Answer 6

In broad terms, I believe the question of when a site comes out of beta should be looked at from a holistic perspective. I don't think it should have to be entirely successful in every category, or meet every specific criteria as has been the case historically, but that it should, synergistically, accomplish an overall goal. Some sites may be stronger in some areas, and weaker in others, than other sites. This should be part of their overall nature—and weaknesses shouldn't be considered in isolation.

In thinking about this, I find it humorous that we have used the word graduate, because I think of graduating from university after having achieved a particular grade point average across all courses that have been taken. In university you can get a poor grade in one course, yet still graduate by making up for that poor score by doing better (overall) in other courses. In short a score below a cut-off value in one area can be overcome by scores above the same cut-off value in other areas.

Here is my answer in terms of a broad approach:

• Each individual category that defines part of a site's health (or success) should contribute to an overall health score (or report card or transcript, if you will).
• Once the site has reached a certain overall health score, it should be eligible to graduate.
• Depending on how this is modelled, it could either graduate as soon as the overall score is reached or it could need to maintain that overall score for a certain period of time.

There are some other considerations:

• The existing criteria for failure should still apply. In other words, there need to be minimums that have to be met for the site to continue even in beta form. If those aren't met, then the site has failed—just as a student would have flunked out of school.
• It seems that there is now also at least one criterion for immediate graduation: having remained in beta for seven years. Whether this is now an established precedent isn't quite clear. It would be nice if it were clearly defined. In terms of the overall health calculation, the number of years a site has been in beta could contribute on some kind of scale—where seven years adds a value equal to what's need for graduation, but where increased longevity before that adds an increasing number to the total score.

Once the specifics of what contributes what amount to a site's total health score have been defined, then each site's current individual (categorized) scores and total score could be shown. This would be a quick way of evaluating exactly where each site is with respect to the minimum score required to stay in existence as well as the target score needed for graduation.

I offer this as a change in the general approach to the idea of graduation. If it should be accepted in theory, then specifics could start to be defined.

Answer 7

It could be basic metrics, like, total questions/answers, just to see grow.

For example, for SOru, in its early stage, we can see grow of q/a in total:

• все сообщения => all posts.
• вопросы => questions.
• ответы => answers.

This metrics are not enough to say about meeting graduation criteria. But it could be used with other metrics.

Answer 8

My starting point would be to have a small number of clear numerical indicators. Ideally, those indicators should be stable in time so that the values do not jump all too much between adjacent days. With a sufficiently small number of indicators there is less need to calculate averages.

I would like to see something like this:

There are three numerical criteria:

1. The number of active users (see below) must be at least 50.
2. The number of good questions (see below) in total must be at least 3000.
3. At least 75 percent of flags are handled within 24 hours.

If these criteria are met and both staff and moderators show green light (see below), the site graduates. (I pulled the bolded numbers out of a hat and they can be adjusted.)

These indicators are discussed below. One of the leading principles behind this thinking is that a site does not have to be fast-paced to be useful and healthy. I don't think 10 questions per day are necessary for a site to function well.

Level of activity

I think the most important measure of maturity is the number of active users. Here is one possible definition of an active user:

A user is considered active if they have completed at least 10 actions (ask, answer, vote, comment, edit, moderate) and has been seen on at least 5 days within the last 30 days.

The numbers 10, 5 and 30 can be adjusted, but I think the order of magnitude should be this. The point of this definition is that an active user is someone who keeps consistently coming back. The exact rate of contribution per user does not have to be sky high; the key is consistent contribution.

If the site is, say, at least six months old and has a good number of active users, some of them are likely to have relatively high reputation. Consistent use often leads to high reputation.

Consistent contributions require that there is something to contribute on. Therefore this requirement will also ensure a decent influx of new questions. A question that many people contribute to by voting, commenting, and answering, is worth more than a question with no attention. Thus I think no separate indicator like number of questions per day is needed.

Size

A site should also be stable enough and should be a valuable repository of knowledge. A simple measure (hopefully) correlating with both is the number of "good pieces of knowledge", which could be defined as follows:

A question is "good" if it has an accepted answer and both the question and the answer have at least score 2. The question must be open and not deleted.

Questions that meet that criterion are likely to be helpful for people searching for information. The score threshold should not be too high. A nice answer to a narrow question on a niche topic will have a hard time attracting tens of votes, but it is just the kind of thing the network exists for — at least from my point of view.

It is possible for a site to change course later in its life, but I find a large bulk of good questions to be a decent guarantee of stability and well-defined scope.

A site doesn't have to be fast-paced to be healthy and useful. I don't think questions per day is all that useful a measure.

Health

A necessary property of a healthy site is the lack of issues. The question is: What indicates problems?

The simplest answer seems to be unhandled flags. I propose measuring this by the median time to handle flags over some period of time. Median rather than average, so that the rare cases where letting a flag stand for days is justified do not contribute absurdly to the indicator. If half the flags are handled very quickly, things are pretty good. Perhaps the "half" could be replaced with another fraction, but not all of the flags.

Human judgement

Finally, human judgement is needed. Perhaps the indicators can be gamed. Perhaps there can be something wrong with a site that is not caught with these indicators. Perhaps the users think that a major scope decision needs to be made before going off beta. Perhaps no flags are unhandled because nobody cares to flag the bad content they see. Perhaps the community doesn't exercise their moderator privileges. Opinions from people are crucial, whether users or staff.

If, say, two staff members and local moderators agree that everything is fine, that should be enough. A meta question can be posted soliciting concerns to allow regular users to voice any issues they see.

The human opinion should be just a simple check, not requiring too much work. The moderators know where their site is going anyway, and staff should quickly spot issues that need addressing. The point is just to have a safeguard in case the algorithm goes awry for whatever reason.

This should be a simple yes/no decision, not a numerical evaluation.

Answer 9

Apart from a well defined scope I would define health as:

• People come and ask questions and get their questions answered. This is the number of questions per day, the number of unique people asking a question, the rate of visitors that ask a question, the rate of questions that get positively scored or accepted answers.
• People come and write answers. This is the number of answers written, the number of unique people writing answers, the rate of visitors that write an answer, the rate of answers that gets positively scored.
• People vote on questions and answers. This may be the rate of non-zero scored questions and answers.
• People visit and read questions and answers. This might be the total number of visits per day, but also the average number of views per question per time duration.
• People do not see garbage or garbage is cleaned up quickly. This could be the ratio of zero or lower scored questions and answers (especially if the total number of votes per question is rather large).

The decision for a possible graduation could be based on a non-linear classifier of all these indicators. Using historical data of functioning and failed StackExchanges (maybe time binned to increase statistics a bit), one could maybe see some limits.

My impression is that problems are not the same everywhere. Some communities may have quality problems (lots of questions, which must be closed), others may not have enough experts to answer (low ratio of positively scored questions that get at least one positive answer), others may just survive but being unable to really grow to their potential (half alive).

Answer 10

Lasting value

Most activities, including votes, happen on new posts in some days after creation. There is a roughly exponential decay, with a half-life of some days.

Beside that, there is another source of the activities, what is roughly time-independent and is typically much slower.

We could approximate the activities as the sum of an exponentially decaying, and a constant function. The second, the constant part, is the lasting value of the posts. The sum of the lasting value of the posts is the lasting value of the site.

Unfortunately, SEDE is probably not enough strong to make such model fitting tasks correctly, but anyways it is not a particularly hard problem.

My proposal is to bind the graduation to the produced lasting value of the sites: measure the lasting value of the current graduated sites as they left beta, and make a simple numerical limit, based on this information.

_{Side effect: publishing this information could also help a lot to motivate people to create content with higher lasting value.}

Answer 11

Moderator Elections

With the current state of Beta, over the last few years. The primary factor for closure has been community involvement sufficient to keep 3 (or so) moderators.

In the last few months we have begun having moderator elections on beta sites. I am strong supporter of recurring elections on Beta sites and further believe that the same can be used a simple measure of community involvement.

A site that has had 2 moderator elections, separated by at least 6 months, with 2 or more qualified candidates running for election is ready to move past Beta.

• No previous election; Set previous count = 0
• Last election was < 6 Months ago; Set previous count = 0
• Last election did not have 2 qualified candidates: Set previous count = 0
• 2 qualified moderators stand for election; Increase count by 1