After less than 24 hours in a fairly closed beta, it's clear that the current voting system is not actually causing the "best" on- and off-topic questions to rise to the top. Out of the ten most-followed proposals, we only found three cases (30%) where the top 10 "on-topics" and the top 10 "off topics" were actually good descriptions of the site. What is the Area 51 team doing to try to improve this?
After watching Area51 for awhile, I have come to the conclusion that the original voting scheme was better. Before, I could simply vote on each question individually. The best (most representative) questions tended to bubble up to the top as many people weighed in on the value of the question.
Now that I have to decide if a question is better than others, and try to keep track of how many votes I have left, I find that there is simply too much cognitive dissonance. It is inhibiting my desire to vote. And apparently that's true for other people as well; as the number of proposals and the restrictions on how you can use your votes both increase, the amount of participation in the evaluation of questions is dramatically decreasing also.
If you are going to attempt to steer the process towards those questions that you like, why bother asking the community?
I totally disagree with this new policy of limiting question examples and votes; this completely kills of the "brainstorming" atmosphere.
It also will discourage people from posting new questions on an already running proposal and/or voting on low-score questions, because nobody will be going to really care about them anyway if the gap with already "established" examples is too great.
Also, while the goal of finding the most relevant on- and off-topic questions is definitely important, I think posting questions freely helps a lot in getting an idea of what the proposed Q&A site will actully become like, thus helping people tailor the proposals better (and also helping the SE team evaluating them properly).
I for me don't see anything wrong in having lots of example questions on a proposal, as long as they're not blatantly obvious or duplicates. I actually find this a lot more useful than keeping them to a minimum and strive to find the top 10 ones. This kind of filtering can be safely done in the second phase, after the proposal has gathered enough followers and examples to get a good view of what can actually come out of it.
I think there's a better way to approach the whole voting issue; please have a look here.
I'm reposting (and slightly editing) this from a different question, which I just deleted the old answer to - I didn't realize that this topic existed, and it's obviously a lot more relevant.
5 total on-topic/off-topic votes is simply absurd. It's already a lot of mental effort to think about some of the more borderline questions (which are the ones that are really important) and reason about whether or not they would belong on their respective sites. Now you're telling us that we have to do this for every question, and that we have to rank them; that's BS.
You're asking for people to do way too much work. And the list of questions is constantly in flux (or at least should be) in the discussion phase, so people have to keep going back and updating their "favorites." Few people have that kind of time or patience - especially when you wear it down by turfing all the decision-making work and critical thinking they've done so far.
I think that the most likely outcome of this change is either: (a) You get a really inconsistent and mostly-useless list of top-rated questions, because people will use up their whopping grand total of 10 votes in 6 seconds and then give up, or (b) You end up with a fraction of the participation that you had originally because everybody hates the system. Or possibly (c) all of the above.
You wave (a) away by saying that once you've found 5 questions for both on and off-topic, your work is done. Is it? That biases the system even more toward early questions, because users won't come back once they've done their voting. You're very likely to end up with an entirely random distribution of votes, per category, and that doesn't help you find the best questions at all.
I can understand your grievance, kind of. But I really think that you need to find a better way, because some of what you're doing will make the situation worse. Maybe you need two tiers of votes - on-topic/off-topic and interesting/boring - basically like how close/reopen votes and upvotes/downvotes work on Stack Exchange sites already. I don't know if that's the right solution, but I'm pretty sure that the one you're apparently about to implement isn't. You just can't ask people to pick 5 and only 5 questions from a list of 150.
Joel: You are the last person I would have expected to make a decision like this. You know that reducing the quantity does not improve the quality. You've written about it on your blog!
Limiting questions is fine, because it doesn't add any further burden to users (if anything it reduces it, there are fewer questions to worry about). And randomizing the sort order is a great idea. But please, think carefully about the long-term effects of your decision to limit votes - not to mention the ill-will you'll receive by trivializing everybody's effort so far.
[I cannot comment because of rep...]
I am increasingly wary about anything that Stackexchange does. You dropped SE 1.0 because it was not working. And, now you are changing the voting rules and asking users to vote again when several of us have spent time thinking through the questions and voting them up or down. I understand that improvement is necessary but the speed with which things change here does not encourage me to spend a lot of time on the proposals or the Stackexchange system.
How do I know that the rules will not change again?
Regarding the limits to votes and questions in #2 and # 6 above, can I assume that is per user per proposal?
I think that the intent of both rules makes sense, and will discourage inane question generation to generate rep points or stalve off boredom. But both numbers seem a tad low, especially the 5-question limit. I think the limiting of the up/down votes, in conjunction with the "Meh" vote, will allow for much more effective selection of the "right" questions (and the wrong ones). Given a strong selection system, you should be more comfortable with a larger sample size, especially since the noise will quickly be Meh'd out of the way.
In my opinion, the "10 most voted" is not enough, and I never considered it to be enough to define the scope.
For me, all these questions are defining the site somehow, but only as a very basic draft. Someone will have to come after to take the actual rules from these questions.
Lesson for later: don't answer Joel spolsky's questions, they are not actual questions.
Edit: I like the current system though, because besides the noise, it is a "brainstorming" atmosphere: you think of a question, you add it, without wondering if it is so important. Same for votes, you don't care if the question is really defining the site, you just say if on or off topic. The new system, in my opinion, will be more "stressing". You have a limited amount of actions, so everything has to be important. Ok, then no more freedom in it, let's only propose a few questions.
But in this case, there was no need for a whole new site. A Meta discussion would have been enough.
What could be better though, would be to leave things like this for the "brainstorming phase", where you determine on and off topic, and add a new "selection" phase, where people will vote if useful or "Meh" on the current batch of questions.
You should look at the questions with the most votes, not those with the highest and lowest score.
People vote when they care or the decision is clear-cut. Clear-cut questions definitely shape the purpose of the proposed site, while emotional/controversial issues (for the beta testers, at least) are the cases likely to be the most interesting and have the biggest impact.
Does looking at the top 20 most-voted questions give you a better picture? Does taking the top 40 and tossing clear cut issues into various buckets (many are likely to be similar) and then looking at the top 20 of what's left give a decent shape?
We are going to implement a few changes that we hope will help.
Feedback is welcome!