We're going to be running a whole lot of tests in the near future (mostly on Stack Overflow) and we'd like to take a moment to set everyone's expectations consistently when it comes to how, what, when and where we'll be announcing these tests as we go.
tl;dr;We've done a pretty good job of announcing tests in the past, but we'd like to make the process a little more formal and consistent across the board.
What kinds of tests are you talking about?
We're talking about A/B tests where a group of people are selected to see some things on one or all of our sites differently than other people see them, as we measure how people interact differently with each scenario.
We're also talking about tests where we make deliberate changes to things and invite a certain group of folks to provide feedback, or situations where we test fixes to stuff in place.
Basically, any scenario where we compare what we think is going to happen against what actually happened.
Okay, what changes?
We want to make certain that:
- We don't ask for attention unnecessarily in the form of announcing tests, especially when the tests only affect anonymous users
- We give logged-in users that will likely see something different due to a test some heads-up if we anticipate the possibility of the test somehow breaking something for them
- We remain very responsive to reports of things going on while we test stuff, even if you just want to know what we're doing. However, we'll of course continue to post something if we detect that we broke something - but you folks generally beat us to that.
- We become more mindful of stuff we can share from tests that we run, and post it more frequently once we've had a chance to really dig into the data
So, we're taking steps to do that.
Why are we clarifying how we're going to be announcing tests?
Because we're going to be running quite a few of them in the very near future, as mentioned. Depending on the results of initial tests, dozens more may kick off, and it's not always possible to get tests running in time and update or post additional notifications.
Tests will, for the foreseeable future, just be running pretty constantly.
Why mention this here, on MSE? You mentioned Stack Overflow
Because it's how we're going to be doing things overall, including any changes to the network. Sure, right now it's Stack Overflow specific, but that won't always be the case, especially when designers are hard at work on stuff.
When does this go into effect?
It mostly has, since it's not that far off from how we were announcing them.
I need to report a bug, or something made me curious
If you run into a glitch that you suspect might be related to testing, just open a bug report or discussion on the corresponding meta site (MSO for Stack Overflow, here for anything in the network) and we'll get on it. Each site has a testing tag that can also be used.
Note: We're talking mostly about things like UI tweaks, changes to text, changes to tooltips, and a myriad of things that can make a big difference in how people actually interact but aren't really all that code impacting at all. In order to get the UX as good as you can, sometimes you just have to move stuff around a bit until you find (and scientifically prove) that you have the right combo, then revisit that every few months as your users change. Totally different than unit and integration tests.