This is a follow-up to my comment on meta SO.

I'm going to start on a chemical plant analogy. To achieve mass production with a new process, one goes through several scaling steps. At each step, the amount of the product increases (simplifying things a little) tenfold: lab, bench, pilot plant, full-scale.

Stack Exchange developers have internal review processes. Number of critics (N) =1. Let's be optimistic and say N=10. Even then, all the critics are biased (that's the consequence of working inside the system).

After passing internal review, SE devs post on meta SO/meta SE and get N=1000 or N=10000 at once:

...and, to crown it all...

This amounts to full-scale responses, and let's be clear - those responses are frank and not really flattering. This frankness provokes a defensive reaction from the devs/CMs, who did not see it coming. What's more, Stack Exchange undermines its reputation by untested projects.

What is missing: an intermediate N=100 feedback phase. An opinion survey that doesn't go full Meta. The exact implementation is unclear at the moment. Could be an e-mail survey for moderators (since they have already signed an NDA).

  • 1
    A big issue with things like these is that your N=100 often leaks it to the N=10000, and then you have 10000 discussing something incomplete and experimental, extrapolating with guesswork, rather than discussing an actual proposal, which is long-term counterproductive.
    – user206222
    Jan 5, 2016 at 10:55
  • It works fine for the iOS app beta for example. @Emrakul Jan 5, 2016 at 10:56
  • @Patrick Maybe I'm misunderstanding the proposal, but I think the iOS beta is not quite the same - it's not a feedback survey as much as it is a bona fide application that has yet to be fully released.
    – user206222
    Jan 5, 2016 at 10:57
  • @Emrakul - that's why the NDA is important. Give the e-mail survey to all SE mods, and reap the collected wisdom. Of course, to maintain it as a long-term resource, SE has to actually listen to the feedback. Jan 5, 2016 at 10:59
  • 1
    Why not just through Meta? @DeerHunter It is the best place to discuss things and the openness to the community is good IMHO. Jan 5, 2016 at 11:00
  • @DeerHunter Moderators aren't a great sample of the general SE population for a couple quirky reasons. But besides that: who pursues NDA violations? Seems a bit much...
    – user206222
    Jan 5, 2016 at 11:02
  • @PatrickHofman - devs are put off by the hostility and revert into "users are dumb we'll do it anyway" mode. Jan 5, 2016 at 11:02
  • I think there is even no need for an NDA. Jan 5, 2016 at 11:02
  • @DeerHunter Proof or it didn't happen! Luckily we have CM's which are the glue between dev and the community. Jan 5, 2016 at 11:03
  • @Emrakul - no, they aren't. Yet they are experienced, have an interest in SE's viability, and have an agreement in place. An alternative would be recruiting 'beta testers', but the self-selection bias is much worse here. Jan 5, 2016 at 11:05
  • @PatrickHofman - that was a general observation from the industry, not necessarily a description of what SE devs do or feel. Jan 5, 2016 at 11:06
  • 4
    Actually, meta kind of is our intermediate N=100 phase. Meta is basically comprised of power users; people who are familiar with the company and its products, many of whom have been around and providing feedback for years. Full-scale feedback incorporates a broader range of personas. We're always open to trying new ways of collecting feedback, so I agree with the overall intention of your post, but I disagree a bit on your premise. :)
    – Laura
    Jan 5, 2016 at 15:16
  • 1
    @Laura - the featured posts are instantly teleported into the N=1000+ area, makes for more difficult scaleup. Also the full openness of Meta may be somewhat awkward for testing new ideas. Jan 5, 2016 at 15:24
  • Meh, I'm not so worried about the awkwardness. When things get awkward, it's usually a sign that we can be better in our communications, or that we actually do need to rethink the idea (or both!). :)
    – Laura
    Jan 5, 2016 at 15:42
  • @Laura - you may want to re-consider in light of recent events. Being negatively reddited is worse than having a tight mailing list with site mods. Jan 15, 2016 at 20:23

2 Answers 2


A place to test new stuff is a good idea.

Stack Exchange is growing fast, and every day there are new features. Some you opt in by Google Docs, some by checking a box in your profile, and others are just activated, without prior notice. There are often complains about why SOI does it this way. Maybe it has worked in the old days, but now most of the users 'just want it to work'. They don't care for a shiny new navigation that is broken, they just want to get an answer to their question.

I would love to see a beta site, like 'beta.stackoverflow.com' where we can test all nice new stuff, but where we can revert to the main site if the beta is broken. This will give users willing to test the ability to do so. Others can just have a working site. New features that are tested can eventually be released to the public.

Feedback can go through the appropriate site-specific meta or Meta SE.

  • 6
    Visualizing the birth of meta.beta.stackoverflow.com ;-) Jan 5, 2016 at 11:51
  • 4
    We did exactly that with the new navigation – it was an opt-in beta with a checkbox under preferences, that you can turn off any time if you want to use the main site. For Documentation, we had an explicit sign-up process and are using a different site to test the concept. We're trying to do more of this where it makes sense, but it doesn't make sense for every project.
    – Laura
    Jan 5, 2016 at 15:18
  • I agree with that Laura, still a unified way to enroll is preferred. And a way to easily switch between the two. (I sometimes wanted to make a screenshot of the old version for a discussion on meta, so I had to log on under another user to get the old version). Jan 5, 2016 at 15:20
  • @Laura - that's why you can get some results with e-mail mod surveys without constructing beta sites. An intermediate canvassing phase cuts on embarrassment... Jan 5, 2016 at 15:32
  • 5
    Yeah, we might be getting to the point where a more unified system of beta testing makes sense, instead of sometimes signing up via a Google form, sometimes checking a box on your preferences, etc.
    – Laura
    Jan 5, 2016 at 15:45

Chat works. It's an informal way to get initial feedback about ideas that aren't secret, and aren't fully formed. If the feedback is negative, then ideas can be clarified and revised quickly. If it really turns out poorly, then you can just say "never mind, thanks for your feedback" with no lasting consequences.

The moderators' chat room could work — moderators tend to be good representatives of the community, and have some experience with politics and group dynamics. The size of the audience is about right: a few dozen moderators are usually active in chat, and up to ~400 can be summoned by word of mouth as appropriate.

  • 1
    ...cough... word of mouth -> word of ping ...cough... Agree with the general idea. Jan 17, 2016 at 5:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .