Be clearer on what a beta site needs to graduate [duplicate]

Question

The beta tab of an Area 51 proposal contains the criteria for considering a beta site healthy:

• 15 questions per day on average
• 90% answered
• 150 users with 200+ rep
• 10 users with 2,000+ rep
• 5 users with 3,000+ rep
• 2.5 answers per question
• 1,500 visits per day

However, as the Robert Cartaino's answer to my question (Are there any clear and objective criteria when the public beta site will graduate? states), this is not enough. There are more fuzzy factors taken into account, before deciding, if the site has a chance of succeeding.

So I propose, that we should sit down and try to objectivise that fuzzy criteria and turn them in some numbers. In fact, this is what happens in the minds of SE team, it's only not yet written.

For example, the question states the problem with the possible lack of users with moderator privileges after changing from beta to graduate site. That means, that the requirements from Area51 are too loose. We need more people with close privileges (>3000) and at least a few ones with access to moderator tools (>10000).

Another factor is the gross size of the site, which makes the critical mass. We could be inspired by Generalist badge criteria, which require at least 40 tags with at least 200 questions in each in order to grand that badge. Maybe 40 tags with 100 questions is too much as graduate criteria, but how about at least 20 with 100 questions?

Anyway, I think there's enough SE sites to sum up the experiences and write out the common problems with beta sites and with sites that should become graduated. Writing clear criteria for graduation would not only make the whole process more transparent, but what's more important, they would become the guidelines for each beta community, saying what should we work on to help our children grow up.

Answer 1

Executive Summary

Creating a "Checklist" of items that guarantee graduation will cause certain members of communities to aim toward achieving the checklist rather than focusing on site quality.

The emphasis should be less on numbers, and more on what defines a healthy community, with numbers as one indicator of whether a community is healthy.

"I know it when I see it" is a very effective tool for handling something as fuzzy as whether a community has the chops to make it.

Gaming the System

How many people review only to the point that they get a badge? How many people will post quick answers and then edit their posts to get the Enlightened badge or a bit more rep? How many people make minor suggested edits for the sole purpose of getting the +2 rep? How many people edited a community wiki post or made a bounty during winterbash just to get a temporary hat?

When you create incentive to game the system, people will game the system. When that's a meaningless hat or a badge, it's not so bad, but when the energy of a new community is focused on achieving arbitrary numbers rather than building the community, the community actually suffers because of it.

^{This is my #1 complaint about the Area 51 procedure -- the arbitrary cutoffs for getting to the next stage, along with the time limitations, encourage people to game the system just to push the process along to get to the actual community building. This encourages appeals to amateurs rather than experts in the sample questions and 'marketing' for the push toward private beta}

The Emphasis Should be on Community Health

No matter how hard you try, numbers will never be able to define who is a good community member, they can only be used as indicators to keep an eye on to see where the community stands so that those community leaders can actually act on improving them.

The users that make a community successful are the ones who focus on the things that actually make a community successful. They define the scope in meta. They help out new users. They curate tags. They make helpful edits. They encourage community behavior. They create relationships with the regular users and make them feel like a part of the community.

Numbers should focus on letting the community know which areas they may want to take a look at and tackle as a community. For instance if meta is like a ghost town, that should be a red flag for the community to act on. On the flipside though, a lot of meta activity doesn't necessarily mean that the meta activity is good. If there are constant discussions of site design, or resolving conflicts between members, or rehashing the same arguments, that doesn't have the same value as meta posts that are defining scope or promoting community moderation. So how do you quantify "Successful Meta"?

"I know it when I see it"

"That's where the truth lies, right down here in the gut. Do you know you have more nerve endings in your gut than you have in your head? You can look it up. I know some of you are going to say "I did look it up, and that's not true." That's 'cause you looked it up in a book. Next time, look it up in your gut. I did. My gut tells me that's how our nervous system works."

- Stephen Colbert

Communities should be graduated when they feel right. They could be growing exponentially, have tremendous participation, but have an ideological rift within the community pulling the same site in two different directions. A community could be in beta for 4 years without seeing the growth that the SE team may expect, but show through their participation that the community is healthy enough to step out on its own two feet. There is no magical formula for what succeeds.

If anything, communities should be given more of an ability to speak directly with the SE Community Managers to discuss the "State of the Beta" to give SE a better view of how healthy the community actually is. The Area 51 stats and the regular site self-evaluations only provide numbers, if anything the focus should be more on the community members who are participating daily and what they feel about the health and strength of the community.