Please consider gathering Meta feedback before the "ship" stage

Question

The blog post "The Loop #1: How we conduct research on the Community team" sheds a little light on how the current process for making changes on Stack Exchange works.

Today, our research process might look something like this for a high-impact, high-cost project.

What's our current research approach?

Strategy (Survey; User interview; Comparative site analysis; Historical context research; Data analysis), Design (User interviews), Build (User interviews), Ship (Survey; A/B tests; Meta feedback)

And later on it says:

Meta feedback. We monitor Meta for bug reports and small usability/copy issues after shipping changes to the site.

Now, as was mentioned in the blog post, this is a marked departure from how things worked in the past.

What was our past research approach?

Strategy (Meta feedback), Design (Meta feedback), Build (Meta feedback), Ship (Experiments)

Before, every stage of the process was simply "Meta feedback". Now, the Meta feedback has been relegated to... bug reports and usability issues.

Please consider gathering Meta feedback earlier in the process.

Meta feedback is important.
Now, as an avid Meta user, I may be biased. However, Meta represents the most engaged users - the ones who have been here for years in most cases, often current or former moderators on the network, and those who are, most importantly, the ones that the site most heavily relies on for moderation. Meta feedback represents the feedback of those most engaged users.

The feedback of your most engaged users, especially on a "high impact" project, is important. Those users are in the best position to tell you how it well affect their use of the site as someone who uses it very regularly, and they may let you know about something that is very important to them but less so to other people.

I'm thinking of cases such as the incident where the Worldbuilding.SE robot, Slartibotfast, was going to be removed due to the design changes to the site. Due to meta feedback from those most invested in the community, the robot was saved and an important part of the site identity was preserved.

Journeyman Geek put it nicely in this post:

I'd also like to point out that, well, the designer is not going to have the same grasp of a site's "uniqueness" as someone who has been on it a while will have.

As I mentioned before, this is a marked departure from how SE worked in the past. You are taking a group who has, for ten years+, been deeply involved in every stage of a change, and suddenly telling them "you're only giving feedback after it's done and we won't be making any big changes because of your feedback. Sorry!"
That's not going to go over well.

Especially in the current climate of the community having broadly lost trust in the company, removing their feedback from any changes is only going to serve to further deteriorate any remaining trust.
On the other hand, including them in the process and working with the community will massively help improve trust and relations between the engaged community and the company.

We, the Meta community, are the people most invested in these sites. The way our feedback is treated should reflect that.

Please consider gathering Meta feedback earlier in the process for making "high-impact" changes.

Answer 1

Thanks for sharing your thoughts with us. I can understand that this is a big departure from our past approach and may exacerbate some of the trust issues that folks here are feeling. I’d like to provide some additional color to this; specifically, how we gather feedback from our most engaged users and why Meta isn't our primary method for doing so.

We do regularly talk to our most engaged users via other research methods (for example, depending on the project, we’ll talk to moderators in 1:1 interviews, target high-rep folks in surveys, etc.). It’s important for us to be hearing from folks most invested in the site, and - especially for high impact projects - we get these perspectives early in the product development process.

The reason that Meta isn't our primary method for talking to our most engaged users is that it's not representative of our most engaged users. For example, only about 10% of our most active answerers are on Meta. So if, for example, we want to hear from answerers, we'll target answerers via other methods rather than put out a general call on Meta and hope that active answerers will respond. Other methods allow us to segment and, more generally, conduct research in a methodologically-sound way that's been vetted by our resident trained researchers.

Answer 2

Online forum discussion is a valid and frequently used source of qualitative data. It has limitations, but can readily be integrated with other methods of data collection. If SE doesn't have anyone on its team that has training in these methods, it may benefit them to get training or explore the literature. There are obvious benefits to community engagement.

Meta users, and, specifically, those users who respond to a specific question, are not a probability sample, but none of the methods described above provide a probability sample either. By integrating meta discussion with other methods, SE would improve coverage as well as community engagement.

In other words, I don't believe there is a good reason from a research methodology perspective to not include "meta feedback" with other data collection methods during the research process. There may be other reasons.

Edit: based on some comments in chat, there seems to be some confusion on my meaning. I appreciate Donna's answer and upvoted it. I'm sure she's expressing her honest belief. I just disagree with her. If she saw my comment before it was deleted, I hope she found my statement of both appreciation and disagreement encouraging and helpful.

Regarding other reasons:

Another staff member with a broader role has stated that meta causes psychological damage. Even if Donna agreed with me that collecting data from meta discussion earlier in the process would be beneficial, I don't think SE would do that. Because of this, I don't think Donna's reason for not collecting data from meta is the reason for this larger business decision, the pivot away from meta.

Answer 3

I'm pleased to say that it looks like SE has taken a step in this direction again. Quoting from New Feedback Tag – [product-discovery]:

We recognize feedback from our meta communities is invaluable and we want to share ideas with you at earlier stages to incorporate your feedback. To avoid confusion and manage expectations, we will do our best to be clear about the type of feedback we’re looking for at each milestone/stage. For example, a feature in the Discovery stage will go through several design iterations before we land on a solution, so we have a lot more room to incorporate feedback before going into Build. Whereas changes to a feature that is at the end stage of Build is costly but may be in need of user acceptance testing.

To clarify what stage a project is in and set the right expectations with you, we plan on introducing new tags (likely 2-3 overall). We’re starting with the mod-only product-discovery tag on all Meta sites. This tag will be used by staff when sharing concepts in the Discovery phase relating to product or configuration changes. In most cases a direction and/or goal has been established, and there has likely been some amount of time invested in research and discovery.

This looks like it's a step in the right direction; they've publicly said that Meta feedback is important to gather before release and are actively working on ways to make that work. It's starting to look like Meta isn't being completely ignored any more and that staff has learned that they need our Meta feedback. So, kudos for that.