Once upon a time, with the question what can be done with Area 51, many users realized that Area 51 doesn't seem to work all of the time, but no one seems to have taken that a step back, in finding what actually makes sites take off. I'm proposing that we take a look at what makes sites successful, and then what should make site successful, and see if we can shrink the distance between the two. Given this approach, I believe we can make the site a better place.
What makes a site successful?
Proof of this:
What should be the metrics?
How could we get there? One thing that I realized when typing this up is that all we need is a few users who are willing to moderate the site in its early stages, and a bunch of people willing to ask questions. It might turn into a free for all early on, but the dust will settle, and the site can be recovered. Science Fiction and Fantasy is living proof of such a site, it was rocky at first, but now stands as a very solid site. All it took was a few people with the power and interest to set the site on the right course.
As for more of a list:
I did a little digging in the Area 51 data:
These are all sites in still beta (80), graduated (38), or which have failed in beta (30). Most of the failed sites had one or two weeks in beta, but a few failed later. Some of the beta site are very near graduation and others are not.
Questions per day is a pretty good proxy for progress toward graduation. It's not the only thing we look at, but it's a very important indicator of the progress a site is making. Compared the the field, graduated sites have a high rate of questions and had more example questions. This isn't surprising, if it's easy to ask questions and lot's of enthusiasm in the proposal, that ought to transfer to the beta.
It might seem strange that the more successful category of site, the fewer average number of Area 51 followers. But that can be explained by the criteria for leaving the definition stage: 60 followers and 40 questions with a score or 10+. The followers requirement never blocks a site from progressing. Sharepoint, with 83 followers, is the closest a site's come to the threshold. Proposals gather more followers when they spend a long time in definition. On the flip side, failed sites seem to fail (in part) because too few followers make it to beta.
More committers, more committers in beta, and better commitment scores tend to be signs of sites that will move quickly to graduation. In this case, we sometimes see proposals stuck in commitment due to low commitment scores and other times because not enough people have signed up. Since we try to launch sites shortly after they reach 100% (notable exception: Stack Overflow (in Portuguese)), more committers is either a sign of runaway momentum (hello, Craft CMS) or struggling to get a commitment score of 1. Of the two, momentum is an indicator of future success and being blocked by commitment score foretells failure.
The best path to future site success follows the lead of Stack Overflow: start with lots of enthusiast and clued-in users, keep the lead up to site creation short, and achieve critical mass from day one. Slow Area 51 progress is a very negative sign. If you compare the rate of example questions in Area 51 to the rate of questions in beta, you can see a general correlation:
You can see at a glance that the more example questions per day in Area 51, the more questions per day you are likely to see in the beta. The biggest outlier, Stack Overflow (in Portuguese), is instructive. First, because it crossed into definition when it was easier, it only had 32 example questions. Then we held it up after it reached 100% in order to find Gabe. That meant it had a record (so far) of 1133 days in Area 51 before being launched. Adding the two factors, our Portuguese SO was launched with the fewest examples per day, but is currently our 17th most active site. Salesforce, a site very near graduation, had just 17 example questions in definition and took a little longer than normal to get enough people with network experience. As a rule, getting a lot of examples and having a short commitment period are the keys to getting a proposal through to graduation.
A note on the other side of the ledger: the Arduino proposal that is an outlier in the bottom right is the proposal that failed in private beta. A substantially better Arduino proposal is now in beta. The successful proposal had more example questions, but a longer gestation. I suspect that reboots work best when they are able to capture some of the audience that participated in the original proposal. That means, generally, to reboot as soon as possible.