I looked through the top sites on Area 51 to see if I thought the new voting system was letting better questions win.

Bottom line: YES! (Here's the data.)

Here’s what I did. I looked at the top 12 sites by follower count. Myanmar doesn’t have many questions, so that left 11 sites.

For each site, I looked at the on-topic list for questions with more than 5 on topic votes, with no more than 10% dissent, and the same for off-topic.

I counted the number of questions that qualified, and then read them all to decide if they are actually good site definitions.

In every single case, we have good site definitions. That’s 100%. That’s up from 30% under the old voting system. Ergo it is my humble opinion that the voting change was stupendously successful in terms of getting better site definitions quickly.

  • 11
    Meh™. "to decide if they are actually good site definitions". I guess that's another way to pass a "Joel Spolsky test". In the end, it's still the opinion of one man only.
    – Gnoupi
    Jun 10, 2010 at 13:07
  • 1
    @Gnoupi: Joel thinks it's the community opinion, and he likes it because it matches his own; but it's only part of the community opinion, because the voting system heavily influenced it. An interesting question is, exactly how and how much...
    – Massimo
    Jun 10, 2010 at 13:11
  • 6
    Exactly. It's my opinion only. I would like to hear your opinion, too, as to whether the winning questions are actually good definitions of the site. Jun 10, 2010 at 13:25
  • 1
    – devinb
    Jun 10, 2010 at 13:30
  • 4
    I like the new voting system. Since you only get 5 on each side to you really have to think about what is a "Great On" and "Great Off" topic question. The old system, everything was off or on, and the great questions never floated to the top and did a really poor job to determine what the site was and more importantly what it was not. Jun 10, 2010 at 13:36
  • 2
    I agree with Joel and @David, this does seem to have worked. Maybe it is because I actually joined after the new voting system was installed, but I certainly have found it very engaging with the site, and it encourages really devoting yourself to the site by checking back and reassigning your votes.
    – Grace Note StaffMod
    Jun 10, 2010 at 13:51
  • @Joel, they surely are; my concern is about what we are losing because of limitations enforced by this system. It's like saying that an absolute monarchy is better than a democracy because there aren't hundreds of congressmen who spend all of their time talking a lot and accomplishing almost nothing; it's obviously more efficient, but is it really better?
    – Massimo
    Jun 10, 2010 at 13:52
  • 1
    I agree with your assessment of the on-topic questions (most fit into the "some" category). As for off-topic questions, I still believe that good examples are few and far between. You need more than 3 or 4 non-trivial examples to explain what the site isn't about.
    – Aarobot
    Jun 10, 2010 at 16:32

3 Answers 3


I do like the new system for the reasons that it was introduced. One byproduct that seems to be happening, though, is that for slower velocity proposals like mine the voting seems to have stalled. Some of the questions on my proposal are getting good votes, and they should be because they are the quality questions that will define the site. It does seem like there is trepidation with the voting, which does follow logically with the new voting system. People seem to be waiting for more questions to get posted before casting their precious votes.

This is just an observation. I'm working on drumming up more followers and hope to see the voting activity increase when we get more. My proposal is fairly niche in comparison to others and not technically focused and as such I realize that attracting followers will take more time and effort than others.

  • 5
    Keep in mind that we haven't told ANYBODY about Area 51--just us weirdos here on meta. Once we announce it widely, through the blogs and on Stack Overflow itself, there will be a LOT more attention to the proposals... by an order of magnitude. Jun 10, 2010 at 14:16
  • 1
    @Joel: Yes, very aware of that. And, from my proposal's perspective, very much looking forward to it! I'm quite happy with the number of weirdos, er, followers that I have attracted from just the beta audience!
    – squillman
    Jun 10, 2010 at 14:36
  • 1
    @Joel: one more thing to point out is that I have done a trivial bit of recruiting for my proposal and have attracted a few followers outside of the meta audience who are not familiar with the SE platform other than what I have told them about it. I'm finding that it might prove difficult coordinating with them as the proposer and have a couple of ideas brewing for feature requests. I'd like to be able to communicate with my proposal's followers better outside of the comments. They're not hitting F5 on the site quite like I do :)
    – squillman
    Jun 10, 2010 at 14:43
  • 1
    @Joel - I doubt that the wide audience will do something else than increasing the already established votes, though.
    – Gnoupi
    Jun 10, 2010 at 15:48
  • @squil, I think the idea is that you are no different from your followers (only by having come up with the idea)... the site would belong to the community as a whole, so why would you need to communicate to followers as if you (as the proposer) were at a different level? (unless I misunderstood how this will work @joel)
    – juan
    Jun 10, 2010 at 17:22
  • @Juan I inherently feel a sense of ownership for it and a responsibility to recruit, which I agree should be shared by any site followers, so "as the proposer" is probably not a good phrase. However, I would like to be able to communicate with those (espeically non-tech) not familiar or used to checking in on a SE-style site/forum outside of the site so as to get them into the habit of checking in. This is especially true for the followers of a proposal since they are the experts on the site. I can foresee attracting many experts who are not part of any online communities. (that make sense?)
    – squillman
    Jun 10, 2010 at 17:31

Joel, how exactly can you evaluate if those questions are the best ones for defining a site amongst those that have been submitted?

This would require comparing them with all the other ones that didn't make it to the top positions.

And by which criteria? Anything a single person can say (including, with all due respect, all SE staff members) is by definition subjective.

In my opinion, having now seen that the new voting system can actually select good questions, the second most important (and interesting) analysis that should be done is:

How many example questions are there that are actually quite good for defining the site, but didn't and/or couldn't get enough votes?

This is what would really tell if the system is too much limiting.

Another good thing to look up (I'd have done it myself if there was a Data Explorer for Area 51) would be:

What trends emerge by comparing question age and question votes?

My impression here is that newer questions, even when actually good, don't get much votes because people already used up their available ones and don't want (or just don't bother) to recast them when new questions are posted.

  • 3
    @Massimo, All Joel is saying is that prior to the voting change, they had identified what they perceived as problems with the definitions and questions pertaining to an idea. They had measurements to backup their perceptions. They decided that they should make modifications to rectify this. They did, and then they measured again, and they found improvement. How is this something to critique? They had measurable goals and they accomplished them. He is simply publishing their success.
    – devinb
    Jun 10, 2010 at 13:28
  • 1
    (-1) for suggesting that the site owners subjective measure of success is less valid than your own subjective measure of success.
    – devinb
    Jun 10, 2010 at 13:29
  • @devinb, where exactly did I mention my measure of success, and/or suggest that it would be more valid than anyone else's?
    – Massimo
    Jun 10, 2010 at 14:43
  • @Massimo "In my opinion, the most important analysis that should be done is: How many example questions are there that are actually quite good for defining the site, but didn't and/or couldn't get enough votes?"
    – devinb
    Jun 10, 2010 at 14:48
  • @devinb, please quote things entirely. That's the most important analysis in order to know if the voting system is too much limiting. I'd hoped it would have beel obvious to everyone that the most important goal of all is finding good questions... anyway, edited it for clarity; could you please remove your downvote?
    – Massimo
    Jun 10, 2010 at 14:57
  • "This is what would really tell if the system is too much limiting" is a statement that attempts to qualify the previous statement, namely your opinion that devinb quoted. It can basically be read as "My opinion of how it should be analyzed is what would really tell if the system is too much limiting". That's most likely where one derives that you're suggesting yours is more valid... because you kinda were in those words.
    – Grace Note StaffMod
    Jun 10, 2010 at 15:02
  • 1
    @ccomet, as I already said (and edited for clarity), I acknowledge the new system can select good example questions; but I have a strong feeling that it limits users too much, thus causing the loss of valuable feedback on proposed sites. I think the best way to evaluate this would be checking if there are good example questions, maybe even better than the most voted ones, which nevertheless got very low votes or none at all (but no NaGEs); this would indeed show the system is too limiting.
    – Massimo
    Jun 10, 2010 at 15:11

The real question is - were those questions already in the proposals BEFORE the reset? Before you claim success I think you need to figure out how many questions of interest (voted up or down) came after the reset. I have a feeling that the unbridled enthusiasm for the new system might be checked with those results...

Do you have a count for new questions?

Also, could we have achieved similar results with just a fix to the BUG about not randomly sorting the answers?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .