What tools and techniques are used to test UI changes?

Question

Stack Exchange sites are used on a lot of different browsers, at screen sizes ranging from huge 4k monitors to small phones, at window sizes ranging from full-screen to 1024px (sometimes smaller), on computers with resources ranging from "I never see a beachball/hourglass" to "I get by, mostly", sometimes with configuration changes for accessibility (like larger fonts or keyboard shortcuts or voice).

That's got to be a bear to test.

In the spirit of Which tools and technologies are used to build the Stack Exchange Network?, I'm curious how SE goes about testing design changes and UI more generally. I seem to recall from the Joel Test in past SE job posts that SE doesn't employ QA folks, and the design team isn't large. How do y'all test the SE design in all its variant environments and still get new work done?

This question overlaps with How TDD-oriented is the SE team?, but that question seems to be more about implementation and automation. You can write unit tests to verify that clicking on a UI thing still works and never notice that the actual display of that thing is off, for instance. I'm really more focused on the design here and the answers might be different.

Answer 1

The TL;DR is it's a time-intensive process, especially when making large changes.

---

Right now the styles through Stack Overflow and Stack Exchange are rather fractured. While they might look fairly close, a lot of them are implemented in different ways and use different CSS classes. This isn't a huge problem when making more localized changes (to just Q&A or just Documentation). Yet when we want to standardize core elements such as colors, type, and form styles — that becomes much harder.

As the OP mentioned, Stack Overflow doesn't employ people who are tasked strictly with QA. Instead that burden is shared across the product team. We have to double-check every potentially affected page manually. Most of the time this isn't a huge deal because we learn fairly quickly what pages are affected; yet there's always a few "I never even knew this existed" moments.

The core tools available to each designer are:

1. A local virtual machine. Designers are expected to do their own front-end coding. Javascript is a bonus. We have local environments so we can test things out locally without blocking production builds.
2. A design system. This is a newer tool for us. As the team has grown, we've seen a need for standardizing and documenting our design system so that the whole team is using the same style. Amongst other projects, we've spent the last 6-8 months creating and implementing this new system. We're not finished yet, but we're close. This system documents all UI elements, UI components, and UX patterns used throughout Stack Overflow and Stack Exchange. We've already standardized colors, buttons, tabs, and soon form/input styles. Most of these updates you hopefully haven't noticed (though a few have).
3. The last thing is the team. We help each other review work and bring up potential issues. Jin Yang and Kurtis Beavers also work a lot with the team to make sure new and old designs are in line with our design styles.

The typical process for a design once it has been approved and is ready to be implemented is this:

1. Designer(s), engineer(s), and product manager come up with a plan.
2. Designers usually do most of the front-end code.
3. Designer(s), engineer(s), and product manager QA the new feature together, reporting visual or coding bugs.
4. Feature is released to the rest of the teams internally for further testing.
5. If more bugs are reported, they're addressed.
6. If everything looks good and we're ready, we push it out to production.

I hope that answers your question.

---

Other questions:

Do you consciously use a mix of VMs (Windows/Mac/Android‌​/iOS/Linux flavors) and browsers as part of this, or does it just fall out naturally (people on your team use different configurations already)?

We all use the same setup for the most part. We all use MacBook Pros or iMacs with a Windows VM with all the major browsers installed locally (on the Mac) and within the VM. For mobile, certain people test using specific hardware. We also ask the team to help out when testing. We haven't had a need to develop Linux or Ubuntu VMs for testing since browsers are fairly standardized at this point except for font rendering.