Once upon a time, with the question what can be done with Area 51, many users realized that Area 51 doesn't seem to work all of the time, but no one seems to have taken that a step back, in finding what actually makes sites take off. I'm proposing that we take a look at what makes sites successful, and then what should make site successful, and see if we can shrink the distance between the two. Given this approach, I believe we can make the site a better place.
What makes a site successful?
- Finding a community of users already within Stack Exchange, who are willing to put a bit of time and effort into the process.
- A short proposal/ commitment time, made possible by #1.
- A clearly defined scope, that can attract a reasonable number of users.
- A scope which there is a large number of questions, which can be reasonably easily found.
- Having advocates for a site tends to make it work better.
- Steady progress towards the goal.
- Maintaining interest in the site
- Some kind of critical mass seems to help, once the site is moving, people get excited about it, and help spread the word.
Proof of this:
- Stack Overflow in Portuguese- Destined to die until a user started referring people, who believed in the site. Now it is well on its way to success.
- Logic Design clearly shows a limited, but good scope, and slow and stead progress towards goals. Activity is shown by the number of questions asked about it.
- Politics is slowly on its way to death, with a somewhat inactive creator and no one really spreading the word about the site.
- Chinese Emigration- Huge interest initially, but not from within SE. Without SE support, it will likely wither and die.
- Smugmug- Huge initial interest, it took over a year to get the site to beta. By that time, the original interest had drained away, leaving the site to wither. The site is alive, but I wouldn't at all be surprised if it died soon.
What should be the metrics?
- A minimum number of users is required.
- How much does the community seem to be forming before the community has officially started?
- Some number of users willing to help police the site.
- Length of time seems important, perhaps length of time since last visit.
How could we get there? One thing that I realized when typing this up is that all we need is a few users who are willing to moderate the site in its early stages, and a bunch of people willing to ask questions. It might turn into a free for all early on, but the dust will settle, and the site can be recovered. Science Fiction and Fantasy is living proof of such a site, it was rocky at first, but now stands as a very solid site. All it took was a few people with the power and interest to set the site on the right course.
As for more of a list:
- I believe that a user should be required to have at least 200 reputation on at least one site prior to opening a proposal on Area 51. Maybe make a minimum 150 reputation or something like that.
- Have an option to commit to "moderate" a site. Have some criteria that shows how they are helping to moderate a site. This wouldn't require asking questions on the site, but maybe get points for closing, downvoting, and participating on meta, in addition to closing questions, to get the commitment back.
I did a little digging in the Area 51 data:
State N QpD views examples Followers in_beta Committers in_beta c_score gestation --------- -- --- ----- -------- --------- ------- ---------- ------- ------- --------- Beta 80 6 11486 65 173 94 246 170 1.03 374 Graduated 38 12 15122 78 157 99 308 220 1.16 127 Failed 30 2 14634 67 192 54 288 157 1.05 372
These are all sites in still beta (80), graduated (38), or which have failed in beta (30). Most of the failed sites had one or two weeks in beta, but a few failed later. Some of the beta site are very near graduation and others are not.
QpD is either the current questions per day statistic or the questions per day when the site left beta (either to graduation or closed).
Committers what each proposal accumulated by the time the site entered beta.
c_score are the number of example questions and the commitment score at the same moment.
in_beta shows how many followers and committers actually joined the beta and
gestation is the time from proposal to the start of the beta.
Questions per day is a pretty good proxy for progress toward graduation. It's not the only thing we look at, but it's a very important indicator of the progress a site is making. Compared the the field, graduated sites have a high rate of questions and had more example questions. This isn't surprising, if it's easy to ask questions and lot's of enthusiasm in the proposal, that ought to transfer to the beta.
It might seem strange that the more successful category of site, the fewer average number of Area 51 followers. But that can be explained by the criteria for leaving the definition stage: 60 followers and 40 questions with a score or 10+. The followers requirement never blocks a site from progressing. Sharepoint, with 83 followers, is the closest a site's come to the threshold. Proposals gather more followers when they spend a long time in definition. On the flip side, failed sites seem to fail (in part) because too few followers make it to beta.
More committers, more committers in beta, and better commitment scores tend to be signs of sites that will move quickly to graduation. In this case, we sometimes see proposals stuck in commitment due to low commitment scores and other times because not enough people have signed up. Since we try to launch sites shortly after they reach 100% (notable exception: Stack Overflow (in Portuguese)), more committers is either a sign of runaway momentum (hello, Craft CMS) or struggling to get a commitment score of 1. Of the two, momentum is an indicator of future success and being blocked by commitment score foretells failure.
The best path to future site success follows the lead of Stack Overflow: start with lots of enthusiast and clued-in users, keep the lead up to site creation short, and achieve critical mass from day one. Slow Area 51 progress is a very negative sign. If you compare the rate of example questions in Area 51 to the rate of questions in beta, you can see a general correlation:
You can see at a glance that the more example questions per day in Area 51, the more questions per day you are likely to see in the beta. The biggest outlier, Stack Overflow (in Portuguese), is instructive. First, because it crossed into definition when it was easier, it only had 32 example questions. Then we held it up after it reached 100% in order to find Gabe. That meant it had a record (so far) of 1133 days in Area 51 before being launched. Adding the two factors, our Portuguese SO was launched with the fewest examples per day, but is currently our 17th most active site. Salesforce, a site very near graduation, had just 17 example questions in definition and took a little longer than normal to get enough people with network experience. As a rule, getting a lot of examples and having a short commitment period are the keys to getting a proposal through to graduation.
A note on the other side of the ledger: the Arduino proposal that is an outlier in the bottom right is the proposal that failed in private beta. A substantially better Arduino proposal is now in beta. The successful proposal had more example questions, but a longer gestation. I suspect that reboots work best when they are able to capture some of the audience that participated in the original proposal. That means, generally, to reboot as soon as possible.