What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

After less than 24 hours in a fairly closed beta, it's clear that the current voting system is not actually causing the "best" on- and off-topic questions to rise to the top. Out of the ten most-followed proposals, we only found three cases (30%) where the top 10 "on-topics" and the top 10 "off topics" were actually good descriptions of the site. What is the Area 51 team doing to try to improve this?

share|improve this question
5  
are you asking "what are they doing?" or "what should they do?" –  Rich Seller Jun 2 '10 at 15:11
14  
My problem is that when I see a question that's obviously off-topic, my knee-jerk response is to vote it off-topic. But from what I understand, the best off-topic questions are the ones that are only barely off-topic. Right? –  mmyers Jun 2 '10 at 15:15
3  
@mmyers: Yes. I have been thinking about it in terms of the FAQ. What are the best 5 questions I would like to see in the FAQ; best on- and off- topic questions to define the site. –  Robert Cartaino Jun 2 '10 at 15:23
    
@mmyers ~ +1 upvote for I concur. Isn't the point of a sample off-topic to help weed out the "unsure if this is OT" questions? –  jcolebrand Jun 2 '10 at 16:05
1  
Don't they work for you? And didn't you just answer your own question? –  Rosinante Jun 3 '10 at 1:11
2  
LOL @ NoP close vote. –  Gnome Jun 3 '10 at 1:23
2  
BTW, why is this discussion happening here instead of meta.stackexchange.com? –  Massimo Jun 4 '10 at 10:57
    
This is the designated place to talk about Area 51 topics, because ultimately meta.stackexchange.com and meta.stackoverflow.com will be the same site (this one) –  Joel Spolsky Jun 4 '10 at 13:43
    
To the close-voters. I realize that the votes are primarily out of protest, but please try to keep this question open so that the discussion can continue. Thanks. –  Robert Harvey Jun 4 '10 at 15:26

8 Answers 8

We are going to implement a few changes that we hope will help.

  1. The default sort order for questions will be randomized. This is the only scientifically proven way to kill the fastest gun in the west problem.

  2. Each user will only have 5 on-topic votes and 5 off-topic votes, so they will have to carefully decide how to allocate them. It is not your responsibility to decide on everything, it's your responsibility to find ten great questions to define the site. Once you've found them, your job is over.

  3. Given that the rules of voting have changed, the only fair thing to do is to discard any votes cast under the old rules. This will also allow us to see if the new system is really working better. (We are not discarding old questions, just old votes.)

  4. There is a third way to vote, Meh. Meh means that you think the question is just not exemplary in any way. Some examples of Meh:

    • The question is too similar to another question already under discussion. In the dog site, there's already a question about Siberian Huskies' diets, and this is just a question about Malamute diets. The question doesn't add anything to the definition of the site.

    • The question is blatantly inappropriate. It's not a question, it's a homework request, it's spam, it's abusive, etc. The question would be deleted ANYWHERE on the Stack Exchange Network, so it's not a good example of an off-topic question because it doesn't illuminate anything about how the site would work.

    • For any other reason, the question is simply not improving the definition of the site.

  5. If a question receives too many Meh votes it is banished to the bottom of the sort order.

  6. Each user, including the original proposer of a site, can only propose 5 questions total. We are trying to discourage proposals with hundreds of sample questions which make it too hard to cast votes responsibly.

  7. To make it easier to see what you voted for, questions that you have voted for will appear at the top of the "random" sort order.

  8. You can change your votes. (This has always been the case).

Feedback is welcome!

share|improve this answer
16  
AAAGGGHHHH, I spent hours deciding what to vote for and voting, and now it's all down the tubes. –  Lance Roberts Jun 2 '10 at 15:22
26  
Voting "Meh" on this answer. –  Gnoupi Jun 2 '10 at 15:26
1  
"Meh" is perfect (otherwise I would have suggested that you additionally vote on the questions quality - which would have needed two clicks) –  Tobias Kienzler Jun 2 '10 at 15:29
2  
"Each user will only have 5 on-topic votes and 5 off-topic votes". Why resetting this, when it's already defined? Questions are already on or off topic, what is missing now is only the "defining" or "Meh". –  Gnoupi Jun 2 '10 at 15:36
5  
I don't understand what you just wrote. –  Joel Spolsky Jun 2 '10 at 15:40
1  
@Gnoupi - agreed, which is what I assumed would be in the second phase (among other things). –  ChrisF Jun 2 '10 at 15:41
    
Limiting proposed questions is the best change here. If you have fewer questions to vote on, there's less need to limit the number of votes each user can cast, and votes will still mean more. I understood not voting under the current system to mean "meh", but on some proposals (Web Applications) I gave up reading through them. –  Gnome Jun 2 '10 at 15:43
2  
@Joel - My English is failing from time to time. I mean that we already know what is on topic, or off topic, after this first phase. Why would we reset this now? All we need now is to select which questions are "defining the site", and which are not. We could even hide the vote count, it doesn't matter now. But voting again for on topic or not doesn't make much sense at this point. –  Gnoupi Jun 2 '10 at 15:45
1  
@The Cat - it hasn't helped Web Applications particularly that phase 1 has been extended (for what ever reason). People want to join in so are still proposing questions when there isn't really any need. –  ChrisF Jun 2 '10 at 15:45
2  
@Lance: If there's a better system out there, I'd rather get it in place now and waste several users' few hours than deal with wasting a ton of users' hours later. –  Gnome Jun 2 '10 at 15:47
4  
@Joel Not everything has to be question/answer on Meta. This might be easier to discuss if you moved the contents of this post into the 'question', so that everyone else can reply with 'answers' instead of comments. –  Kyle Cronin Jun 2 '10 at 15:54
1  
I wonder if this could be evaluated per proposal. Questions (and votes) on the gaming site are quite spot on –  jmfsg Jun 3 '10 at 2:00
1  
@TheHurt - I'm pretty sure that's one "f" too much. –  Gnoupi Jun 4 '10 at 11:05
1  
You of all people, Joel, should know that limiting the quantity does not improve the quality. And now you're about to go and piss off a lot of people by killing all of the decision-making work they've done. –  Aarobot Jun 4 '10 at 13:21
6  
@Joel If you can take your retarded idea and and propose SO, and through the process end up with what SO is today, I will stop calling your idea retarded. –  TheHurt Jun 4 '10 at 14:25

In my opinion, the "10 most voted" is not enough, and I never considered it to be enough to define the scope.

For me, all these questions are defining the site somehow, but only as a very basic draft. Someone will have to come after to take the actual rules from these questions.


Lesson for later: don't answer Joel spolsky's questions, they are not actual questions.

Edit: I like the current system though, because besides the noise, it is a "brainstorming" atmosphere: you think of a question, you add it, without wondering if it is so important. Same for votes, you don't care if the question is really defining the site, you just say if on or off topic. The new system, in my opinion, will be more "stressing". You have a limited amount of actions, so everything has to be important. Ok, then no more freedom in it, let's only propose a few questions.

But in this case, there was no need for a whole new site. A Meta discussion would have been enough.

What could be better though, would be to leave things like this for the "brainstorming phase", where you determine on and off topic, and add a new "selection" phase, where people will vote if useful or "Meh" on the current batch of questions.

share|improve this answer
1  
If I ask "what is the team doing" (not "what should the team do") that's your clue that maybe I have some inside information :) –  Joel Spolsky Jun 2 '10 at 15:24
    
You are correct. This is only a first step to defining the site--establish the basic ground rules. Detailed discussion comes after. –  Robert Cartaino Jun 2 '10 at 15:25
5  
I was working on this assumption. The initial phase gathered a whole bunch of questions which we voted on. Once enough people had "followed" and there were enough questions the second phase would sift through the questions to actually define the site. –  ChrisF Jun 2 '10 at 15:29
3  
@Joel: Formulate the answer at the bottom of the question body; cut it before submitting the question and paste as an answer immediately afterwards. :) –  Gnome Jun 2 '10 at 15:40

I totally disagree with this new policy of limiting question examples and votes; this completely kills of the "brainstorming" atmosphere.

It also will discourage people from posting new questions on an already running proposal and/or voting on low-score questions, because nobody will be going to really care about them anyway if the gap with already "established" examples is too great.

Also, while the goal of finding the most relevant on- and off-topic questions is definitely important, I think posting questions freely helps a lot in getting an idea of what the proposed Q&A site will actully become like, thus helping people tailor the proposals better (and also helping the SE team evaluating them properly).

I for me don't see anything wrong in having lots of example questions on a proposal, as long as they're not blatantly obvious or duplicates. I actually find this a lot more useful than keeping them to a minimum and strive to find the top 10 ones. This kind of filtering can be safely done in the second phase, after the proposal has gathered enough followers and examples to get a good view of what can actually come out of it.


Edit:

I think there's a better way to approach the whole voting issue; please have a look here.

share|improve this answer

I'm reposting (and slightly editing) this from a different question, which I just deleted the old answer to - I didn't realize that this topic existed, and it's obviously a lot more relevant.

5 total on-topic/off-topic votes is simply absurd. It's already a lot of mental effort to think about some of the more borderline questions (which are the ones that are really important) and reason about whether or not they would belong on their respective sites. Now you're telling us that we have to do this for every question, and that we have to rank them; that's BS.

You're asking for people to do way too much work. And the list of questions is constantly in flux (or at least should be) in the discussion phase, so people have to keep going back and updating their "favorites." Few people have that kind of time or patience - especially when you wear it down by turfing all the decision-making work and critical thinking they've done so far.

I think that the most likely outcome of this change is either: (a) You get a really inconsistent and mostly-useless list of top-rated questions, because people will use up their whopping grand total of 10 votes in 6 seconds and then give up, or (b) You end up with a fraction of the participation that you had originally because everybody hates the system. Or possibly (c) all of the above.

You wave (a) away by saying that once you've found 5 questions for both on and off-topic, your work is done. Is it? That biases the system even more toward early questions, because users won't come back once they've done their voting. You're very likely to end up with an entirely random distribution of votes, per category, and that doesn't help you find the best questions at all.

I can understand your grievance, kind of. But I really think that you need to find a better way, because some of what you're doing will make the situation worse. Maybe you need two tiers of votes - on-topic/off-topic and interesting/boring - basically like how close/reopen votes and upvotes/downvotes work on Stack Exchange sites already. I don't know if that's the right solution, but I'm pretty sure that the one you're apparently about to implement isn't. You just can't ask people to pick 5 and only 5 questions from a list of 150.

Joel: You are the last person I would have expected to make a decision like this. You know that reducing the quantity does not improve the quality. You've written about it on your blog!

Limiting questions is fine, because it doesn't add any further burden to users (if anything it reduces it, there are fewer questions to worry about). And randomizing the sort order is a great idea. But please, think carefully about the long-term effects of your decision to limit votes - not to mention the ill-will you'll receive by trivializing everybody's effort so far.

share|improve this answer

After watching Area51 for awhile, I have come to the conclusion that the original voting scheme was better. Before, I could simply vote on each question individually. The best (most representative) questions tended to bubble up to the top as many people weighed in on the value of the question.

Now that I have to decide if a question is better than others, and try to keep track of how many votes I have left, I find that there is simply too much cognitive dissonance. It is inhibiting my desire to vote. And apparently that's true for other people as well; as the number of proposals and the restrictions on how you can use your votes both increase, the amount of participation in the evaluation of questions is dramatically decreasing also.

If you are going to attempt to steer the process towards those questions that you like, why bother asking the community?

share|improve this answer
1  
There does need to be more ability to remove marginal questions from the system, but the only way you're going to be able to do that is to give people more votes. –  Robert Harvey Jun 4 '10 at 15:21
    
One more example of a time when I wish I could vote 100+... I've just given up participating on A51. I don't think that was the intent of the changes, but that is the effect for me. –  tim Jun 8 '10 at 16:12

You should look at the questions with the most votes, not those with the highest and lowest score.

People vote when they care or the decision is clear-cut. Clear-cut questions definitely shape the purpose of the proposed site, while emotional/controversial issues (for the beta testers, at least) are the cases likely to be the most interesting and have the biggest impact.

Does looking at the top 20 most-voted questions give you a better picture? Does taking the top 40 and tossing clear cut issues into various buckets (many are likely to be similar) and then looking at the top 20 of what's left give a decent shape?

share|improve this answer
    
"Most votes" is not a reliable measurement, as shown by the most voted questions on the trilogy. Funny questions, or obvious ones will be on top, for sure. So for sure another "round" of sifting is necessary. –  Gnoupi Jun 2 '10 at 15:53
    
@gno: The questions with the most votes on SO are those people feel the most emotional about. The only exceptions are questions linked on blogs/reddit/podcast or at the top of the highest voted page. For example, "Hidden features of C#" has surely been linked elsewhere and people can easily "care"/"feel emotional" about the general language they use everyday. That C# is the most popular language on SO and this question is the most voted shouldn't surprise anyone. Emotional != best. –  Gnome Jun 2 '10 at 16:04
1  
Ok, so we are most likely saying the same thing, without fully understanding each other :] –  Gnoupi Jun 2 '10 at 16:20

Question:

Regarding the limits to votes and questions in #2 and # 6 above, can I assume that is per user per proposal?

Comment:

I think that the intent of both rules makes sense, and will discourage inane question generation to generate rep points or stalve off boredom. But both numbers seem a tad low, especially the 5-question limit. I think the limiting of the up/down votes, in conjunction with the "Meh" vote, will allow for much more effective selection of the "right" questions (and the wrong ones). Given a strong selection system, you should be more comfortable with a larger sample size, especially since the noise will quickly be Meh'd out of the way.

share|improve this answer
1  
Yes, per user per proposal. –  Joel Spolsky Jun 2 '10 at 17:23

[I cannot comment because of rep...]

I am increasingly wary about anything that Stackexchange does. You dropped SE 1.0 because it was not working. And, now you are changing the voting rules and asking users to vote again when several of us have spent time thinking through the questions and voting them up or down. I understand that improvement is necessary but the speed with which things change here does not encourage me to spend a lot of time on the proposals or the Stackexchange system.

How do I know that the rules will not change again?

share|improve this answer
3  
AFO is beta, isn't it? Things change much more quickly during beta. As towards other changes, view the track record on Meta as regarding SO, SF, and SU. There have been lots of changes, but almost all of them have been improvements. Start at status-~. –  Gnome Jun 3 '10 at 1:24
3  
Someone from IT may expect what 'beta' means but the proposals will eventually have people (non-IT crowd) who would not expect the rules to change in the middle of the game. Some stability is essential for the system to get traction. In any case, every system will have issues and we cannot keep changing the rules every now and then. By the way, my understanding of 'beta' is that the software may have bugs but I do not expect the basic functionality of the software to change 'drastically'. –  Anon Jun 3 '10 at 1:46
4  
I suppose it could be better labeled as a Beta, but change should be expected in a Beta - that's its purpose. (Plus, the whole process is really a launching zone for a product, rather than a product itself.) And while I agree that too much change in the end-state version undermines buy-in, don't we all want it to work? I know I'd rather see it adapt its model to one that is likely to be more effective. While that's frustrating at first ("I have to go vote again"), it really represents the site adapting to reflect what we're telling it we need and what will work for us. –  Jaydles Jun 3 '10 at 16:14
1  
Just a note for you, Anon, on Meta it only takes 1 reputation to comment anywhere. –  Grace Note Jun 4 '10 at 11:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .