First some background: In November of 2022, I proposed a site called Programming Language Design and Implementation, which has slowly progressed through the Area 51 process, and is now set to enter public beta in one week. This has given us some insight into where the process does and doesn't work well, and since there seems to be some discussion of changing how sites are created, I figured I'd throw in PLDI's experiences and how they could help guide the refinement of the site creation process.
What works, and what doesn't
Some things worked well:
- Definition provided a good opportunity to refine the scope of our site
- Definition and Commitment ensured we had a properly large community before the site went public
Some things didn't:
- Hitting Definition's follower requirement was trivial, but reaching the required number of 10+ score questions was agonizing, and took many months of strategizing and coordinating to redistribute votes most effectively
- By the time we reached major milestones, it had been long enough since the last one that many seemed to forget about the proposal (our initial commits happened far far slower than our initial follows in Definition stage, for example)
- The suggested stats for Beta are unrealistic for most actual Q&A sites
Voting in Definition is basically useless for its intended purpose. The five vote maximum feels far too low; only being able to upvote five questions, when we had over 110, felt impossible to use for actual feedback, and instead they were just used tactically to push questions to, and not above, +10. Additionally, since downvotes are unlimited but upvotes are severely limited, great questions would linger at or below +1 or +2, unless they were posted early enough that they had a chance of hitting +10.
I believe voting should work much differently, so that it can actually be used to refine and moderate the scope of the site. Allowing, for example, 10 votes/week would allow continued moderation efforts, without going overboard. This would also mean that the proposal won't linger for months with 200 people who already followed, waiting to pick up a few more votes, as for every week that goes by, more voting would be possible.
The point of the high-scoring questions rule shouldn't be to require some high number of people to interact with the proposal; that's the job of the follower/committer count. The voting is to ensure the scope of the site is established, so allowing more votes wouldn't compromise the effectiveness of the site proposal process in this regard.
Dividing the process into Definition, which requires followers, and Commitment, which requires nearly identical committers, feels unnecessary, and introduces several problems. Since these seem to serve roughly the same purpose (ensuring enough people are interested), I propose merging them into one stage.
Doing both Definition and Commitment at the same time (I'll call this stage Alpha) would have a number of advantages. For example, instead of discussion of the site's scope abruptly stopping halfway through the proposal process, it can continue up until Beta. Personally, I'd forgotten tons of consensus we'd established during Definition, since we'd spent so much time in Commitment rarely discussing it. If we went straight from Definition to Beta (and without so much dead-time due to the question score requirement), it'd be a more seamless transition into Beta.
A second reason is that the redundant Commitment stage causes confusion. People who already followed and contributed to the site in Definition may not realise they need to come back and click another button, and I had several people I tried to refer go back to the Definition tab and follow there, instead of committing, out of confusion.
This is a smaller one, but sites rarely get "excellent" in all five categories, and in some, like questions/day, are often deeply into "needs work". Changing these to more realistic targets, and maybe adding things like moderation stats (e.g., time things spend in review queues), would give a more accurate picture of the health of betas.
Why not abandon Area 51?
"Community Stakeholders", as proposed for the Prompt Engineering site, likely won't be sustainable as a means of creating sites. For all of Area 51's flaws, it does ensure one thing very well: that there are enough people interested, and that deeply invested "stakeholders" are working behind the scenes to ensure the site succeeds. Area 51 fosters this process naturally, and filters out site proposals doing it ineffectively, without anyone needing to be appointed.
If the process of pushing a site proposal through to Beta were improved upon, it would be a very effective way for the best proposals, and the ones with dedicated stakeholders working to organize a community, to naturally prove themselves. Doing this artificially will only move the process of selecting stakeholders and filtering proposal ideas to being human-controlled, potentially introducing inefficiency or bias.