22

We're going to be running a whole lot of tests in the near future (mostly on Stack Overflow) and we'd like to take a moment to set everyone's expectations consistently when it comes to how, what, when and where we'll be announcing these tests as we go.

tl;dr; We've done a pretty good job of announcing tests in the past, but we'd like to make the process a little more formal and consistent across the board.

What kinds of tests are you talking about?

We're talking about A/B tests where a group of people are selected to see some things on one or all of our sites differently than other people see them, as we measure how people interact differently with each scenario.

We're also talking about tests where we make deliberate changes to things and invite a certain group of folks to provide feedback, or situations where we test fixes to stuff in place.

Basically, any scenario where we compare what we think is going to happen against what actually happened.

Okay, what changes?

We want to make certain that:

  • We don't ask for attention unnecessarily in the form of announcing tests, especially when the tests only affect anonymous users
  • We give logged-in users that will likely see something different due to a test some heads-up if we anticipate the possibility of the test somehow breaking something for them
  • We remain very responsive to reports of things going on while we test stuff, even if you just want to know what we're doing. However, we'll of course continue to post something if we detect that we broke something - but you folks generally beat us to that.
  • We become more mindful of stuff we can share from tests that we run, and post it more frequently once we've had a chance to really dig into the data

So, we're taking steps to do that.

Why are we clarifying how we're going to be announcing tests?

Because we're going to be running quite a few of them in the very near future, as mentioned. Depending on the results of initial tests, dozens more may kick off, and it's not always possible to get tests running in time and update or post additional notifications.

Tests will, for the foreseeable future, just be running pretty constantly.

Why mention this here, on MSE? You mentioned Stack Overflow

Because it's how we're going to be doing things overall, including any changes to the network. Sure, right now it's Stack Overflow specific, but that won't always be the case, especially when designers are hard at work on stuff.

When does this go into effect?

It mostly has, since it's not that far off from how we were announcing them.

I need to report a bug, or something made me curious

If you run into a glitch that you suspect might be related to testing, just open a bug report or discussion on the corresponding meta site (MSO for Stack Overflow, here for anything in the network) and we'll get on it. Each site has a tag that can also be used.

Note: We're talking mostly about things like UI tweaks, changes to text, changes to tooltips, and a myriad of things that can make a big difference in how people actually interact but aren't really all that code impacting at all. In order to get the UX as good as you can, sometimes you just have to move stuff around a bit until you find (and scientifically prove) that you have the right combo, then revisit that every few months as your users change. Totally different than unit and integration tests.

  • 1
    This smells like testing in production, which is bad. How would you feel being on a plane where they are testing (in flight!) a new version (or just some tiny bugfix) of the automatic pilot software? Or how do you think banking systems are being tested? And what do the SE auditors think / say about this, I can't believe they would allow this to happen. – Pierre.Vriens Jul 1 '17 at 10:36
  • 8
    @Pierre.Vriens I'd assume that this for testing user response to new features, rather than looking for bugs. – thesecretmaster Jul 1 '17 at 14:21
  • 1
    @thesecretmaster as if for new features it is impossible that they come with new bugs, right? It doesn't matter what the "change" is about, you do not, repeat not, test in production ... – Pierre.Vriens Jul 1 '17 at 17:29
  • 4
    @Pierre.Vriens We're talking mostly about things like UI tweaks, changes to text, changes to tooltips, and a myriad of things that can make a big difference in how people actually interact but aren't really all that code impacting at all. In order to get the UX as good as you can, sometimes you just have to move stuff around a bit until you find (and scientifically prove) that you have the right combo, then revisit that every few months as your users change. Totally different than unit and integration tests. – Tim Post Jul 3 '17 at 12:32
  • 3
    @Pierre.Vriens: Most major websites test things like user response in production. And I haven't the slightest idea why you would assume there would be "auditors" that would be concerned with that. – Nathan Tuggy Jul 3 '17 at 15:05
  • 1
    @Pierre.Vriens actually most mature continuous deployment pipelines want to test stuff in production. You don't have your Simian Army running around on a meager test system ;) – Helmar Jul 3 '17 at 19:49
  • Good to ensure a consistent level of announcement. People hate to be treated like experiment subjects. – Nemo May 19 '18 at 7:02

You must log in to answer this question.

Browse other questions tagged .