How can we ensure the site creation process consistently targets the right level of expertise?

Question

With the Economics site about to close, I've been thinking about the level of expertise that seemed to be forced on it. I can't know for sure whether it contributed to the failure of the site, but I still think it's important to discuss and consider for future site proposals.

When I committed to the Economics proposal, it seemed fairly clear that it was a site for enthusiasts. Not complete beginners, but not limited to experts either. The tag line is currently "Beta Q&A site for economists and graduate-level economics students" but if I remember correctly it wasn't so specific when I committed. Regardless, there's much else to indicate it was not an experts-only proposal:

• By my count, only 62 / 226 committers selected the option to show that they were experts or graduate-level students (though others left comments to that effect). There were a lot of beginners and enthusiasts committed and, IMO, not enough experts to sustain an expert-level site. What were the beginners supposed to contribute, anyways?

• 7 of the top 10 example questions are extremely basic (general reference or nearly guaranteed to be covered in ECON 101/102). Of the other three, one is asking for a definition and another for a book recommendation; I see only one good question of the lot.

• Plenty of beginner questions were asked when the site was launched.

• A Research Economics proposal existed at that time and was not merged. An apt comparison was made between that site and Theoretical Physics, and the plain Economics proposal and the plain Physics site. Physics includes "questions of all levels", and it would make sense if Economics did as well. Having one proposal for experts and another specifically for those experts doing academic research seems ridiculous to me.

Joel made a meta post about how we should only ask expert questions during the beta (screenshot for when that link goes down). My response partially dealt with one of my own questions, which is no longer of consequence. The relevant portion was:

If [the standard is higher than merely "beyond ECON 101"], I'll have to ask the SE team to cancel my commitment to this site, because this is not what I foresaw. The definition questions are honestly pretty terrible, but the immense gulf between them and your ideal makes the definition phase completely worthless. I've been through several beta launches but nothing has struck me as quite this pointless.

I think this is really the crux of the matter. If we're not going to make enthusiast proposals, I can live with that. But an expert proposal needs to sign up experts who are going to frame the site in terms of that expertise. We can't make a viable site for experts by forcing out half the committers and expecting the rest to make a great site with no foundation in a good definition phase.

Joel also said:

Nothing hurts a site more than making the experts think that the site is stupid or intended for beginners. I can't think how many people who would have made those sites great but did not participate because well-meaning private-beta participants filled the site up with a bunch of really basic questions.

Don't let this happen here. If you aren't an economist, that's fine. Wait until after the private beta.

To which I responded:

Does that really make sense? The experts will enjoy crap questions as long at they're not posted during the private beta? That's bizarre. There might be some sort of lock-in effect — experts who get active early may be more reluctant to leave later — but there are at least two counters to that:

• The experts will feel suckered. They can come here and read what you're saying, you know. Attempting to trick them into believing this site is for experts only to allow the beginners to inundate the site later can't go over well.

• The initial experts aren't enough. There are two few people on the site right now for it to be viable long-term, so we have to grow. To grow, we need to be attractive to new users. If we want those new users to be experts, then we will always need to avoid the "stupid beginner questions". If not, then the site's not a site for experts anyways so why do we care about them at all?

If the site really is for experts, it needs to cater to them consistently. This mixed message nonsense makes the site unusable for anyone. And aren't the biggest sites in the network the ones that don't cater only to PhDs? Didn't SO allow terrible questions once and still manage to get huge and attract the likes of Eric Lippert? (I'm not advocating bad questions, just pointing out that the logic isn't 100% sound here.)

My conclusions from this and TL;DR:

If we want to make sites for experts, we have to do so from the ground up. We can't co-opt enthusiast proposals and make them something they're not. We can't ignore the composition of committers to a proposal — expert sites need to be founded by experts. We can't tell half our committers that they can't ask the questions they defined the site to be about. We can't tell our experts that this site is for expert questions and then flood the site with beginner questions after some set date.

I think part of dealing with this is closing and merging off-base proposals earlier, which I know the SE folk have been doing well lately and I applaud them for that. The other part is making sure we don't change proposals so significantly during their definition, or at least notifying the committers/followers of the changes. We should also require committers to expert proposals to say "Yes, I am an expert" instead of proceeding to launch an expert site full of beginners.

Other ideas, discussion, and disagreement are welcome.

Answer 1

How can we ensure the site creation process consistently targets the right level of expertise?

sorry for the TL;DR answer

The short answer is… using our current process, you cant.

The biggest failure mode of our current site-creation process is that we can define the scope, and we can measure if there are enough users, but the process fails to detect if the founding members are — for lack of a better word — "qualified" to create the site.

The earliest community of a new site typically becomes the core community who will go on to drive the content and — quote-unquote — "expertise" of the site. So if that early group wallowed too deeply into the "beginner" or "just curious" zone, the site itself would likely never attract an expert core audience. See what happened to No Artificial Intelligence in Area 51. That's just an archetypal example of the larger problem.

I appreciate your suggestions about where to look for a solution, but believe me… we have already embarked on a much more wholly-considered approach that goes far beyond fixing Definitions or attempting to fix the Commitment process. Ptooey, good riddance. The future does not include a contrived patched-up Area 51 process — trading it in instead for a a much more thoughtful, educational, and engaging process to build sites. This is still all very early in the drawing-board stages, so I don't have any details yet.

Answer 2

I completely agree that you can't change the scope and expected level of a site after the Area 51 process is nearly finished, that just can't work. If we want only expert-level questions asked, the site creation process needs to make that clear.

But I don't think restricting sites to expert-level questions only is a good idea, I don't like the idea of separating the sites according to the level of the questions. At the same time I have to admit that a large number of laymen questions hurt our chances to attract more experts. At the point where a site looks like it's mostly populated by amateurs, it will have a hard time to convince the experts to spend their time there. I don't mean only questions here, amateur answers can hurt the first impression even more.

I don't think easy questions always hurt the impression experts will have of the site, as long as the easy question gets a good answer I suspect that most experts won't think badly of the site. It gets problematic when the easy, or worse, the lazy questions make up the majority of the site. A site that seems to be mostly populated by amateurs just won't look that attractive to the experts. You also need to have a critical mass of experts to ensure that bad answers get weeded out, that they're downvoted and their flaws pointed out in comments. If the site fails to attract that critical mass of experts, it won't be making the internet a better place, as bad information will stay unchallenged.

One problem with the current process is that its inherently more likely to attract interested amateurs, and not experts. A large part of the user base for a new site is recruited from existing sites, I strongly suspect that those are less likely to be experts. The SE network provides new sites with a large and easy-to-reach userbase, but that userbase is not an expert audience.

The problem I see is that the few experts that are there in the beginning of a new site can be easily drowned out by the larger number of amateurs. The experts are likely fewer in numbers, and in my experience also tend to ask fewer questions. This becomes far less of a problem once enough expert population is attracted to the site, but it can be problematic in the beginning.

One idea would be to start out a new site with a far more restrictive scope, aiming for the expert audience and relax the scope somewhat when the site gets off ground. The impression I get from the SE team is that one idea for the new Area 51 process is to make it easier to start new sites, and quicker to fail them. This very early stage should be directly targeted towards experts, and it should give a far better way to judge if the proposal attracted the right number of experts than the "I'm an expert"-checkbox currently in place. Changing the scope later this way is certainly problematic, and I'm not sure if it's really a good idea, but it might help to attract experts and set an early example that strongly influences how the site will develop later.

My post might seem pretty hostile to amateurs, but I strongly believe that on a healthy SE site the easy, laymen questions can coexist with the expert-level questions and that both sites benefit from that.

Answer 3

You're right - the site should have a clear target audience from its inception. To me, the word "economics" alone is extremely general. What if Bob wants help with taxes? Is that OK too?

So yes, the old saw about "stitch in time" applies. Define what the site will be, then find the right title.

But every site inevitably gets newbie questions. I'm not sure how the decision was made here -it seems that site closure rules are somewhat strict . Is it costly to keep sites with low activity open(why not allow a laissez-faire policy til action is needed)?

By the way, I think the way "dead" sites are stored for anybody curious is user-unfriendly. Why not have it live somewhere as a static page, rather than having to download a zip-file and figure out how to view it?