11

This is a follow-up to my comment on meta SO.

I'm going to start on a chemical plant analogy. To achieve mass production with a new process, one goes through several scaling steps. At each step, the amount of the product increases (simplifying things a little) tenfold: lab, bench, pilot plant, full-scale.

Stack Exchange developers have internal review processes. Number of critics (N) =1. Let's be optimistic and say N=10. Even then, all the critics are biased (that's the consequence of working inside the system).

After passing internal review, SE devs post on meta SO/meta SE and get N=1000 or N=10000 at once:

...and, to crown it all...

This amounts to full-scale responses, and let's be clear - those responses are frank and not really flattering. This frankness provokes a defensive reaction from the devs/CMs, who did not see it coming. What's more, Stack Exchange undermines its reputation by untested projects.

What is missing: an intermediate N=100 feedback phase. An opinion survey that doesn't go full Meta. The exact implementation is unclear at the moment. Could be an e-mail survey for moderators (since they have already signed an NDA).

  • 1
    A big issue with things like these is that your N=100 often leaks it to the N=10000, and then you have 10000 discussing something incomplete and experimental, extrapolating with guesswork, rather than discussing an actual proposal, which is long-term counterproductive. – Aza Jan 5 '16 at 10:55
  • It works fine for the iOS app beta for example. @Emrakul – Patrick Hofman Jan 5 '16 at 10:56
  • @Patrick Maybe I'm misunderstanding the proposal, but I think the iOS beta is not quite the same - it's not a feedback survey as much as it is a bona fide application that has yet to be fully released. – Aza Jan 5 '16 at 10:57
  • @Emrakul - that's why the NDA is important. Give the e-mail survey to all SE mods, and reap the collected wisdom. Of course, to maintain it as a long-term resource, SE has to actually listen to the feedback. – Deer Hunter Jan 5 '16 at 10:59
  • 1
    Why not just through Meta? @DeerHunter It is the best place to discuss things and the openness to the community is good IMHO. – Patrick Hofman Jan 5 '16 at 11:00
  • @DeerHunter Moderators aren't a great sample of the general SE population for a couple quirky reasons. But besides that: who pursues NDA violations? Seems a bit much... – Aza Jan 5 '16 at 11:02
  • @PatrickHofman - devs are put off by the hostility and revert into "users are dumb we'll do it anyway" mode. – Deer Hunter Jan 5 '16 at 11:02
  • I think there is even no need for an NDA. – Patrick Hofman Jan 5 '16 at 11:02
  • @DeerHunter Proof or it didn't happen! Luckily we have CM's which are the glue between dev and the community. – Patrick Hofman Jan 5 '16 at 11:03
  • @Emrakul - no, they aren't. Yet they are experienced, have an interest in SE's viability, and have an agreement in place. An alternative would be recruiting 'beta testers', but the self-selection bias is much worse here. – Deer Hunter Jan 5 '16 at 11:05
  • @PatrickHofman - that was a general observation from the industry, not necessarily a description of what SE devs do or feel. – Deer Hunter Jan 5 '16 at 11:06
  • 4
    Actually, meta kind of is our intermediate N=100 phase. Meta is basically comprised of power users; people who are familiar with the company and its products, many of whom have been around and providing feedback for years. Full-scale feedback incorporates a broader range of personas. We're always open to trying new ways of collecting feedback, so I agree with the overall intention of your post, but I disagree a bit on your premise. :) – Laura Jan 5 '16 at 15:16
  • 1
    @Laura - the featured posts are instantly teleported into the N=1000+ area, makes for more difficult scaleup. Also the full openness of Meta may be somewhat awkward for testing new ideas. – Deer Hunter Jan 5 '16 at 15:24
  • Meh, I'm not so worried about the awkwardness. When things get awkward, it's usually a sign that we can be better in our communications, or that we actually do need to rethink the idea (or both!). :) – Laura Jan 5 '16 at 15:42
  • @Laura - you may want to re-consider in light of recent events. Being negatively reddited is worse than having a tight mailing list with site mods. – Deer Hunter Jan 15 '16 at 20:23
6

A place to test new stuff is a good idea.

Stack Exchange is growing fast, and every day there are new features. Some you opt in by Google Docs, some by checking a box in your profile, and others are just activated, without prior notice. There are often complains about why SOI does it this way. Maybe it has worked in the old days, but now most of the users 'just want it to work'. They don't care for a shiny new navigation that is broken, they just want to get an answer to their question.

I would love to see a beta site, like 'beta.stackoverflow.com' where we can test all nice new stuff, but where we can revert to the main site if the beta is broken. This will give users willing to test the ability to do so. Others can just have a working site. New features that are tested can eventually be released to the public.

Feedback can go through the appropriate site-specific meta or Meta SE.

  • 6
    Visualizing the birth of meta.beta.stackoverflow.com ;-) – Shadow The Dragon Wizard Jan 5 '16 at 11:51
  • 4
    We did exactly that with the new navigation – it was an opt-in beta with a checkbox under preferences, that you can turn off any time if you want to use the main site. For Documentation, we had an explicit sign-up process and are using a different site to test the concept. We're trying to do more of this where it makes sense, but it doesn't make sense for every project. – Laura Jan 5 '16 at 15:18
  • I agree with that Laura, still a unified way to enroll is preferred. And a way to easily switch between the two. (I sometimes wanted to make a screenshot of the old version for a discussion on meta, so I had to log on under another user to get the old version). – Patrick Hofman Jan 5 '16 at 15:20
  • @Laura - that's why you can get some results with e-mail mod surveys without constructing beta sites. An intermediate canvassing phase cuts on embarrassment... – Deer Hunter Jan 5 '16 at 15:32
  • 5
    Yeah, we might be getting to the point where a more unified system of beta testing makes sense, instead of sometimes signing up via a Google form, sometimes checking a box on your preferences, etc. – Laura Jan 5 '16 at 15:45
4

Chat works. It's an informal way to get initial feedback about ideas that aren't secret, and aren't fully formed. If the feedback is negative, then ideas can be clarified and revised quickly. If it really turns out poorly, then you can just say "never mind, thanks for your feedback" with no lasting consequences.

The moderators' chat room could work — moderators tend to be good representatives of the community, and have some experience with politics and group dynamics. The size of the audience is about right: a few dozen moderators are usually active in chat, and up to ~400 can be summoned by word of mouth as appropriate.

  • 1
    ...cough... word of mouth -> word of ping ...cough... Agree with the general idea. – Deer Hunter Jan 17 '16 at 5:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .