With the Economics site about to close, I've been thinking about the level of expertise that seemed to be forced on it. I can't know for sure whether it contributed to the failure of the site, but I still think it's important to discuss and consider for future site proposals.
When I committed to the Economics proposal, it seemed fairly clear that it was a site for enthusiasts. Not complete beginners, but not limited to experts either. The tag line is currently "Beta Q&A site for economists and graduate-level economics students" but if I remember correctly it wasn't so specific when I committed. Regardless, there's much else to indicate it was not an experts-only proposal:
By my count, only 62 / 226 committers selected the option to show that they were experts or graduate-level students (though others left comments to that effect). There were a lot of beginners and enthusiasts committed and, IMO, not enough experts to sustain an expert-level site. What were the beginners supposed to contribute, anyways?
7 of the top 10 example questions are extremely basic (general reference or nearly guaranteed to be covered in ECON 101/102). Of the other three, one is asking for a definition and another for a book recommendation; I see only one good question of the lot.
Plenty of beginner questions were asked when the site was launched.
A Research Economics proposal existed at that time and was not merged. An apt comparison was made between that site and Theoretical Physics, and the plain Economics proposal and the plain Physics site. Physics includes "questions of all levels", and it would make sense if Economics did as well. Having one proposal for experts and another specifically for those experts doing academic research seems ridiculous to me.
Joel made a meta post about how we should only ask expert questions during the beta (screenshot for when that link goes down). My response partially dealt with one of my own questions, which is no longer of consequence. The relevant portion was:
If [the standard is higher than merely "beyond ECON 101"], I'll have to ask the SE team to cancel my commitment to this site, because this is not what I foresaw. The definition questions are honestly pretty terrible, but the immense gulf between them and your ideal makes the definition phase completely worthless. I've been through several beta launches but nothing has struck me as quite this pointless.
I think this is really the crux of the matter. If we're not going to make enthusiast proposals, I can live with that. But an expert proposal needs to sign up experts who are going to frame the site in terms of that expertise. We can't make a viable site for experts by forcing out half the committers and expecting the rest to make a great site with no foundation in a good definition phase.
Joel also said:
Nothing hurts a site more than making the experts think that the site is stupid or intended for beginners. I can't think how many people who would have made those sites great but did not participate because well-meaning private-beta participants filled the site up with a bunch of really basic questions.
Don't let this happen here. If you aren't an economist, that's fine. Wait until after the private beta.
To which I responded:
Does that really make sense? The experts will enjoy crap questions as long at they're not posted during the private beta? That's bizarre. There might be some sort of lock-in effect — experts who get active early may be more reluctant to leave later — but there are at least two counters to that:
The experts will feel suckered. They can come here and read what you're saying, you know. Attempting to trick them into believing this site is for experts only to allow the beginners to inundate the site later can't go over well.
The initial experts aren't enough. There are two few people on the site right now for it to be viable long-term, so we have to grow. To grow, we need to be attractive to new users. If we want those new users to be experts, then we will always need to avoid the "stupid beginner questions". If not, then the site's not a site for experts anyways so why do we care about them at all?
If the site really is for experts, it needs to cater to them consistently. This mixed message nonsense makes the site unusable for anyone. And aren't the biggest sites in the network the ones that don't cater only to PhDs? Didn't SO allow terrible questions once and still manage to get huge and attract the likes of Eric Lippert? (I'm not advocating bad questions, just pointing out that the logic isn't 100% sound here.)
My conclusions from this and TL;DR:
If we want to make sites for experts, we have to do so from the ground up. We can't co-opt enthusiast proposals and make them something they're not. We can't ignore the composition of committers to a proposal — expert sites need to be founded by experts. We can't tell half our committers that they can't ask the questions they defined the site to be about. We can't tell our experts that this site is for expert questions and then flood the site with beginner questions after some set date.
I think part of dealing with this is closing and merging off-base proposals earlier, which I know the SE folk have been doing well lately and I applaud them for that. The other part is making sure we don't change proposals so significantly during their definition, or at least notifying the committers/followers of the changes. We should also require committers to expert proposals to say "Yes, I am an expert" instead of proceeding to launch an expert site full of beginners.
Other ideas, discussion, and disagreement are welcome.