Some suggestions for reworking Area 51

Question

First some background: In November of 2022, I proposed a site called Programming Language Design and Implementation, which has slowly progressed through the Area 51 process, and is now set to enter public beta in one week. This has given us some insight into where the process does and doesn't work well, and since there seems to be some discussion of changing how sites are created, I figured I'd throw in PLDI's experiences and how they could help guide the refinement of the site creation process.

What works, and what doesn't

Some things worked well:

• Definition provided a good opportunity to refine the scope of our site
• Definition and Commitment ensured we had a properly large community before the site went public

Some things didn't:

• Hitting Definition's follower requirement was trivial, but reaching the required number of 10+ score questions was agonizing, and took many months of strategizing and coordinating to redistribute votes most effectively
• By the time we reached major milestones, it had been long enough since the last one that many seemed to forget about the proposal (our initial commits happened far far slower than our initial follows in Definition stage, for example)
• The suggested stats for Beta are unrealistic for most actual Q&A sites

Suggested improvements

Voting

Voting in Definition is basically useless for its intended purpose. The five vote maximum feels far too low; only being able to upvote five questions, when we had over 110, felt impossible to use for actual feedback, and instead they were just used tactically to push questions to, and not above, +10. Additionally, since downvotes are unlimited but upvotes are severely limited, great questions would linger at or below +1 or +2, unless they were posted early enough that they had a chance of hitting +10.

I believe voting should work much differently, so that it can actually be used to refine and moderate the scope of the site. Allowing, for example, 10 votes/week would allow continued moderation efforts, without going overboard. This would also mean that the proposal won't linger for months with 200 people who already followed, waiting to pick up a few more votes, as for every week that goes by, more voting would be possible.

The point of the high-scoring questions rule shouldn't be to require some high number of people to interact with the proposal; that's the job of the follower/committer count. The voting is to ensure the scope of the site is established, so allowing more votes wouldn't compromise the effectiveness of the site proposal process in this regard.

Committing

Dividing the process into Definition, which requires followers, and Commitment, which requires nearly identical committers, feels unnecessary, and introduces several problems. Since these seem to serve roughly the same purpose (ensuring enough people are interested), I propose merging them into one stage.

Doing both Definition and Commitment at the same time (I'll call this stage Alpha) would have a number of advantages. For example, instead of discussion of the site's scope abruptly stopping halfway through the proposal process, it can continue up until Beta. Personally, I'd forgotten tons of consensus we'd established during Definition, since we'd spent so much time in Commitment rarely discussing it. If we went straight from Definition to Beta (and without so much dead-time due to the question score requirement), it'd be a more seamless transition into Beta.

A second reason is that the redundant Commitment stage causes confusion. People who already followed and contributed to the site in Definition may not realise they need to come back and click another button, and I had several people I tried to refer go back to the Definition tab and follow there, instead of committing, out of confusion.

Beta metrics

This is a smaller one, but sites rarely get "excellent" in all five categories, and in some, like questions/day, are often deeply into "needs work". Changing these to more realistic targets, and maybe adding things like moderation stats (e.g., time things spend in review queues), would give a more accurate picture of the health of betas.

Why not abandon Area 51?

"Community Stakeholders", as proposed for the Prompt Engineering site, likely won't be sustainable as a means of creating sites. For all of Area 51's flaws, it does ensure one thing very well: that there are enough people interested, and that deeply invested "stakeholders" are working behind the scenes to ensure the site succeeds. Area 51 fosters this process naturally, and filters out site proposals doing it ineffectively, without anyone needing to be appointed.

If the process of pushing a site proposal through to Beta were improved upon, it would be a very effective way for the best proposals, and the ones with dedicated stakeholders working to organize a community, to naturally prove themselves. Doing this artificially will only move the process of selecting stakeholders and filtering proposal ideas to being human-controlled, potentially introducing inefficiency or bias.

Answer 1

A few ergonomic improvements I'd like to see:

• People should be able to flesh out questions during the Definition phase beyond just a title, even if they're not then shown to anyone else.
  • People should also be able to prepare answers, or drafts of answers, even if they're not shown to anyone.
• Important discussions happen on Area 51 Meta, and then get discarded: it would be good to be able to migrate the relevant ones to the private beta's meta site.
• We should be able to write /help/on-topic during the Definition phase. Currently we get to write one sentence of it.
• Seconding your "beta metrics" point: we should have more realistic stats for what counts as a "healthy community" – especially surrounding the questions per day metric. A site with 1.1 really good questions per day for several years is a perfectly healthy site. (The only site that could be getting 10 questions per day and isn't, to my knowledge, is Pets Stack Exchange.)

Answer 2

Overall, your points are good and I do agree with them, but one aspect is missing and that is the overall curation of the network, which is something that is not well implemented today. And Programming Language Design and Implementation is a good example of this problem, but by no means the only example.

Engagement metrics are, from my understanding, a concern for the company and the CM team. Metrics such as views, returning visitors, new questions, new answers, voting, new users participating multiple days, and more are all on the decline. Fracturing the network doesn't help. I can speak from experience when I say that more core area of expertise is spread across four (or five, depending on exactly what you count) sites on the network - it's difficult, if not impossible, for a busy professional to be active on all of them concurrently.

I'm not going to say these processes don't exist, because they do. However, as a moderator on one of the sites with significant overlap (and an overlap that was heavily minimized), I can say that not only were the moderator teams not reached out to, but the "power users" were not invited through a Meta post, either. When the company is looking at engagement metrics, one of my first questions is what they are doing to make sure that scopes of sites overlap with the experts' knowledge.

Now, don't take this to mean that Programming Language Design and Implementation shouldn't exist. It very well should. Perhaps this is another instance of poor communication between the companies and the existing communities. It's definitely something that should be built into a new site launching process, though.

Answer 3

I've mused about the workings of the current process a bit on the Area 51 Discussions site, but to address your major points directly here:

Voting

I don't think giving people more votes here has any benefit. To quote myself:

At this point, the definition phase only really aids us in identifying what the proposal is meant to be. It's kind of like a game of Whose Line Is It Anyway where the questions are made up and the votes don't matter.

So long as we can identify that a proposal doesn't entirely overlap other sites and can come up with solid questions that don't deserve to be closed, the votes on those questions are rather pointless and don't aid in the process in any way. We don't need to give users more votes, we need to just eliminate voting (almost) entirely. It's a worthless activity that achieves nothing. So long as the score of the question remains positive and it's not closed, it should count. Who cares how much above 1 the score is.

Commitment

If the voting process were simplified in this way, the definition process wouldn't really exist at all and would become even more pointless. It'd be so easy to speed through the definition process with a valid community behind it that it might as well be combined in the commitment phase. We'd just be asking people to click a button and then come back soon and click another button.

Combining them would also alleviate manual work on behalf of the Community Team. Voting fraud wouldn't irreversibly advance a proposal to the commitment phase where the only recourse is shutting it down and having them start over. It also means that the Community Team would have a chance to look through example questions and close and/or comment on ones they think wouldn't be appropriate, giving the community a chance to explain themselves.

I don't know their current process, but for a long time while I was involved the Community Team didn't ever do evaluations of proposals until after definition was complete, which left them without much say if something went wrong in that phase. That's especially problematic if they do identify a problem. Shutting down a proposal after users invested so much time getting through that phase is quite the slap in the face, which is why I always tried to shut down obviously problematic proposals as quickly as possible - don't let users get their hopes up and waste their time!

Beta metrics

Honestly, the solution here is to just not display the suggestions at all. They are completely made up and never were indicative of a successful site. There probably are no numbers indicative of a successful site. They're interesting numbers to show, but showing them beside an unachievable or unrealistic goal is detrimental. We should just drop the descriptions next to each stat and only keep the blue boxes with the stats themselves.

---

The theme here is simplification. A lot of these extended processes existed in the past to make it harder for bad proposals to make it to the end goal without failing. But we've vastly simplified the process to interject ourselves more directly in shutting down proposals that:

• Don't come to us with a pre-existing community ready to build the site.
• Fail to meet basic thresholds within the first three days.
• Fail to maintain any activity over a week-long period.

These simple checks get rid of a lot of the proposals that get suggested on the site. The long-form process has been rendered obsolete by the years-long learning experience we've gone through that has shown us the basic needs for a new site, and most of Area 51 is not needed anymore.

If it helps get developer movement to simplify the site so it can run again, I would even recommend ripping out the entire reputation and badges system there as well. Of the privileges that remain there, most are awarded at 50 reputation (just change that threshold to has verified their email address, done) and the other two important ones are awarded at 150 (verified email + network association bonus, done). The other exist for the sake of existing and can be removed. There we go, reputation system obsolete.

Ultimately, letting proposals linger for 4 months in each phase is still a bit too long. Our goals for Area 51 should be speeding along proposals that have legitimate communities and enthusiasm behind them without much obstacle, and putting up road blocks for people who are just throwing ideas at a wall and hoping they will stick.