What is the SE team's response to community feedback on the new Area 51 voting system?

Question

When the initial Area 51 voting system's problems started to become apparent and the SE team decided to (quite drastically) change it, they said "Feedback is welcome!" (actual citation from Joel Spolsky here).

And feedback it was. There has been quite a lot of it. Mainly here and here, although I also consider this to certainly be relevant.

The feedback, of course, varies; but it has been basically of four main types (from worst to best):

• "Oh sh*t!"
• "This is just wrong."
• "I tried it and it has some serious problems."
• "We'll eventually get accustomed to it, but it isn't exactly wonderful."

The most relevant point here is, there hasn't been even a single enthusiastic response to the new voting system. "It's just great!" is the only feedback that nobody actually gave.

I don't know what the SE team thinks about this, but such a negative feedback would have made most people re-examine very carefully the changes they had just introduced.
The problem is, I don't know what the SE team thinks because there has been no feedback at all from them: they changed the voting system, asked for feedback from the community, then simply disappeared. There has been responses (mainly from Jeff Atwood) to bug reports and feature requests, so someone certainly is reading posts here... but nobody directly addressed the issue of how this new voting system has been received by the user community.

Ok, I'm just going to ask what's actually going on here.

What's the SE team's position about this?
Do they acknowledge the new system has been mainly received with rejection and frustration?
Are they going to do anything at all about this?
And, basically, do they really care about the community feedback?

I'm looking for official answers here. Even "we love it and we'll defend it with our own lives" would be a lot better than just no answer at all. "We're working on it" would at least be a sign that, well, they're working on it.

I sincerely hope this post does not look like a flame; this is definitely not what I mean. But I really think some explanation is needed here.

Answer 1

Nobody is ignoring your feedback. Part of gathering feedback is LISTENING to what people have to say. This is a community-driven forum for feedback and discussion. Thousands of people have a say here. If I issued instant verdicts in on every recently-opened issue, that would effectively end any sort of on-going contribution.

These aren't bug reports where you respond with "ah yes, confirmed." These are usability issues. You release something, you watch how people use it, you gather feedback, you tweak where you can, you redesign where you have to. There's no instant gratification pill to "your voting system sucks." Saying "we're working on it" all the time is redundant and noise. We're working on it. Consider every vote on one of your suggestions an "I agree with you." Consider all the recent activity here, here, here, and here as someone stepping aside from their work and saying "we're working on it."

To answer your question, here's a quick overview from the guy who gets just about every stitch of mail that comes in about Area 51 and reads just about every single message posted to meta.

Limiting the number of questions:

Overwhelmingly positive feedback. Users didn't like the glut of questions asked. Most felt that users were asking questions to get reputation rather than making the proposal better. Users didn't want to wade through all that noise. Many people feel we should allow less than five questions each (surprising).

I like making questions a precious resource. You ask your best questions or you won't get the votes. I prefer that to the down-voting mechanisms (taking away reputation) as a way of encouraging good content in Area 51. I'm losing that argument.

Limiting the number of votes:

Initially, somewhat neutral to negative. Some users wanted more votes, some wanted to vote on everything, some liked the change. The more people realize they aren't voting on each question but selecting their favorite from the nominations, the more they feel we are on the right track.

I agree but I think the interface is awkward. If users just scanned the list and picked "good" questions, the system would work. But, understandably, users feel compelled to select the five perfect questions and optimize for that activity. The current interface doesn't support that. We need a better way to manage use question review and selection

The voting reset:

Whether you agreed with the reasons for the change or not, most people supported the decision to issue a do-over to get things right early on rather than fighting with bad data later.

Most of the down-side was with regard to wasted effort. In any beta, it always says "Don't use this for production work." It's nice that we can get working systems out of this in the end (these aren't practice proposals). I think Google ruined what it means to be in a beta test.

The voting interface:

Not a lot of feedback on that one. What little feedback we got was mostly negative.

Most of my concerns about the voting interface revolve around the selection/list management (discussed above). I'll have to come back later and update this or talk about this issues in another post.

I hate these epic posts so let's call it. Anything else I have to add is better handled as individual posts. Now back to your regularly scheduled program.

Answer 2

First of all, I should say that we really appreciate all the great feedback we're getting here. We're continually reviewing it, and will incorporate it into future iterations.

I want to explain, from my point of view, what our goals are with this change just so you know how we evaluate if it's working or not.

Why is there a "definition" phase in the first place?

Let me work backwards.

In the crucial early minutes of a new Q&A site's existence, it's likely to get blogged and tweeted and a lot of people come by to check it out.

Some of those people post questions.

If they get answers quickly, they will think the site is amazing and come back. And the site will take off like a rocket. If they don't get answers, they'll say, "oh, another dumb community, nobody goes there," and leave.

Thus, a new site works best if we can get a lot of people coming in at the same time on the first day answering each other's questions and building momentum. If the same number of people dribble in over the course of two weeks, not nearly as many questions get answered and those that do take a lot longer.

So, we created a commitment phase... a phase where we gather a list of people for two purposes:

1. So that we can email them all when the site opens, bringing in a critical mass of people all at the same time.
2. So that we can gauge whether there are enough people interested in Q&A on a given topic.

Now--still working backwards here--in order to gather names on the petition to create a site, you need a proposal to show people. The proposal has to tell them what the site would be about, so that they can decide if they would participate.

We could write all the proposals ourselves, but that would be communism. Thus, we attempted to come up with a system whereby the community itself could collaboratively create proposals that explain what a site might be about.

The first system we came up with had a serious bug in that the sort order in which questions were presented was not randomized. As a result, the very act of voting was effectively meaningless, because the first few on- and off-topic questions to be proposed ALWAYS won. They were not the best on- and off-topic questions, either, they were just first. This was, essentially, a bug in democracy. We had to fix it.

And we had another bug in the old system, whereby people had as many votes as they wanted, so people who spent three times as long going through proposals had three times as many votes. This too was a bug. We had to fix it.

So far, the new system does not seem to have either of these bugs, so I think, by our metric, it is actually working. By limiting the number of votes, we send a message that the community's job is to select a few great questions, not to rate all questions. People appear to be doing that, and, most importantly, the top on-topic questions and the top off-topic questions are now much better than they were under the old system.

What we care about is getting proposals that do a good job of describing the site to someone who comes in at the commitment phase and needs to decide whether they want to commit, and I think we're a little bit closer to that than we were when we started.

Now, I realize that we're in an early beta here, so things are changing, and that's annoying, and sometimes it feels like amateur hour, because, well, we're amateurs. We all strongly feel like it's worth the time now to get this right, spend an extra week or two fine tuning the process, because our goal is to make the Internet a better place to get expert answers to questions, and that's a big goal, and one day there will be millions of people using these sites that we define today, and when we think back to this little beta with a few hundred participants, and think of what an amazing thing we built out of that little beta, we'll be amazed.

Answer 3

Answering here to Robert's post, because comment space is too limited.

1. First of all, as I already said, no flame was intended. Everyone knows you're working on it (and is grateful for that). But, as I agree with you that saying "we're working on it" continuously would be just noisy, I also think that sometimes a little more communication just helps. Particularly on controversial issues like this one.
2. I never disagreed with limiting the questions each user can submit. I actually completely agree with that one. But they were already limited to 5 example questions per user before the Big Change, so this isn't the issue here.
3. Au contraire, I think limiting the user's ability to interact with the site has the only effect of frustrating them; do you honestly think S[OFU] would work better if users could only upvote one answer per question? Do you think there would be any need for that?
4. I have no issue with the voting reset per se; this is beta, things like that can happen. But something went bad in this specific case: it was just too rushed. I don't know how much time the SE team had been thinking about it, but we users just saw "we'll change the voting system because the current one doesn't work well" on day one, and the change being done on day two; there has been no time for us to provide any feedback, and for you to evaluate it. Gathering feedback before such an invasive change could probably have allowed you to implement a better system... and possibly without the need to delete existing data (f.e. that wouldn't have been required with the solution I proposed). Also, it would not have created that awful feeling of being ignored some of us experienced.
5. About the voting interface, I already expressed my thoughts, as many others did.

Answer 4

Joel, I have to respectfully disagree with the premise that some well-defined and voted on set of questions will allow me to decide if I want to join a firearms SO site or not or a bicycling SO site or not or an organic farming site.

I'll join them no matter what the limitations or questions are. Why? Because I am interested in those topics and I participate in them every day in the real world and like to discuss them.

It is not the PEOPLE who are going to be different on the site(s) - it is the QUESTIONS.

I know I won't convince you of this, but it will be the same set of people REGARDLESS of this definition phase...

All that this change will likely do is postpone the launch and perhaps have very precisely defined rules and questions. I think the same sorts of questions will get voted up and you are going to have the same users.

I hope it all turns out well, but frankly I find it a bit tedious. I understand you and your team really do like to do things the right way. But I don't see how this is going to really make a difference. If people like Astrology then they are going to try an astrology site. Moving the margins of acceptability slightly based on some sample questions is not likely to have much effect.

Answer 5

I like the new system.

Answer 6

Answering to Joel here, same problem as above with comment space.

Joel, I understand your goal to find the most representative questions for a new site, and I totally agree with it. But I think there's a point you're missing here, or, to be more precise, a point that you're disregarding in favor of pursuiting this goal; I'm talking about the crucial brainstorming phase that follows a proposal and defines what the new site's actual content will be.

When a new proposal is made, stating "this site will talk about X" is clearly not enough; the proposal author will have his/her own ideas, and every potential user will have his/her own ones, too; and there will be things on which various people just will not agree, or even potential topics they will not even have thought about, before some other person mentioned them.

The initial brainstorming phase is exactly about this: defining the site boundaries; this happens through the posting of example questions, through the voting and through the discussions people have about them, in the community process you mentioned. This is what the "on-topic"/"off-topic" votes need to accomplish: reaching a common agreement on the site's topic. If this isn't done properly, the site's life (if it can even be born) will be quite difficult, and endless debates will arise when borderline questions will be actually posted (and they will be).

What the current voting system limits heavily is exactly this: the ability of potential site users to freely discuss about what is on-topic or off-topic. Besides usability problems with the system (already discussed endlessly), you just can't classify all questions as on- or off-topic: you simply don't have enough votes to do that, so everyone will just focus on the most relevant ones, because the very act of classifying them is in fact an upvote. So lots of questions will just not be classified, leaving many gray areas, which should instead be defined before the site goes live.

I'm totally with you on the issue of finding the most representative questions; but I think defining the site boundaries is at least as important, if not even more.

This is exactly the main reason behind my proposal of decoupling on-topic/off-topic votes from up/down votes. This way, people would be free to talk as much as they want about what is or is not on-topic, but only the most useful and representative questions will get upvoted.

Besides that, I completely agree with you on the random order issue; also, I could agree with limiting upvotes, although I don't think this would be necessary, with the system I'm proposing; but I really think the two types of vote should be distinctly separated.

---

Update:

A little clarification. I never said thet everything should be defined in this phase, and/or that every possible question should be analyzed before finding the most relevant ones; I acknowledge this is (and will probably always be) an ongoing process, which even on S[OFU] isn't finished yet; trying to complete it before a site even goes live would be crazy.

But, at the same time, I think this should at least partly be done before the most relevant questions can even be found. Defining, at least in a broad sense, what the site boundaries are is a prerequisite in order to do anything else. The details will of course have to be defined during the site Beta... but that will have to leverage the on-topic/off-topic example questions which are discussed in the first phase.

Basically, my approach to a site proposal is "first define, or at least discuss in depth, what is on-topic and what is not; then (or meanwhile) find the most relevant example questions and start the Beta"; instead, it seems the SE team's approach is "first find the most relevant example questions and start the Beta, then define clearly what's on-topic and what's not". I think the second one is a little... awkward.

---

Update:

I'm really curious about the issue described in the last paragraph; is this the way the SE team really think things should work?