126

The blog post "The Loop #1: How we conduct research on the Community team" sheds a little light on how the current process for making changes on Stack Exchange works.

Today, our research process might look something like this for a high-impact, high-cost project.

Transcription below

What's our current research approach?

Strategy (Survey; User interview; Comparative site analysis; Historical context research; Data analysis), Design (User interviews), Build (User interviews), Ship (Survey; A/B tests; Meta feedback)

And later on it says:

Meta feedback. We monitor Meta for bug reports and small usability/copy issues after shipping changes to the site.

Now, as was mentioned in the blog post, this is a marked departure from how things worked in the past.

Transcript below

What was our past research approach?

Strategy (Meta feedback), Design (Meta feedback), Build (Meta feedback), Ship (Experiments)

Before, every stage of the process was simply "Meta feedback". Now, the Meta feedback has been relegated to... bug reports and usability issues.

Please consider gathering Meta feedback earlier in the process.

Meta feedback is important.
Now, as an avid Meta user, I may be biased. However, Meta represents the most engaged users - the ones who have been here for years in most cases, often current or former moderators on the network, and those who are, most importantly, the ones that the site most heavily relies on for moderation. Meta feedback represents the feedback of those most engaged users.

The feedback of your most engaged users, especially on a "high impact" project, is important. Those users are in the best position to tell you how it well affect their use of the site as someone who uses it very regularly, and they may let you know about something that is very important to them but less so to other people.

I'm thinking of cases such as the incident where the Worldbuilding.SE robot, Slartibotfast, was going to be removed due to the design changes to the site. Due to meta feedback from those most invested in the community, the robot was saved and an important part of the site identity was preserved.

Journeyman Geek put it nicely in this post:

I'd also like to point out that, well, the designer is not going to have the same grasp of a site's "uniqueness" as someone who has been on it a while will have.

As I mentioned before, this is a marked departure from how SE worked in the past. You are taking a group who has, for ten years+, been deeply involved in every stage of a change, and suddenly telling them "you're only giving feedback after it's done and we won't be making any big changes because of your feedback. Sorry!"
That's not going to go over well.

Especially in the current climate of the community having broadly lost trust in the company, removing their feedback from any changes is only going to serve to further deteriorate any remaining trust.
On the other hand, including them in the process and working with the community will massively help improve trust and relations between the engaged community and the company.

We, the Meta community, are the people most invested in these sites. The way our feedback is treated should reflect that.

Please consider gathering Meta feedback earlier in the process for making "high-impact" changes.

  • 51
    looking at SEs (non-) treatment of meta and meta users I'd conclude: that ship has sailed. pun intended. – vikingosegundo Dec 11 '19 at 16:11
  • 5
    You read my thoughts. The new post notices A/B test (the 50% one) was so bad, full of bugs and you couldn't even get out of it. – CaldeiraG Dec 11 '19 at 16:13
  • 31
    I commend you for your tone in writing this. Most other meta users would justifiably have assumed a much more frustrated tone, to put it lightly. This blog post is a big smack in the face to us. – user-2147482600 Dec 11 '19 at 16:38
  • 7
    You might want to contrast the new flow against the old one as they put forth in the blog. When you realize Meta went from Step 3 to Step 4... that's eye opening. I also guess that posts like this are no longer true. – Machavity Dec 11 '19 at 16:47
  • Comments deleted: A pretty long thread about getting invited for surveys, which culminated in this Q&A. – Tinkeringbell Dec 12 '19 at 9:52
  • @Machavity "I also guess that posts like this are no longer true. " Indeed, it's no longer true, but it's also from 2017 and soon there is 2020. Things change over time. – Trilarion Dec 12 '19 at 11:21
  • 2
    I foresee some posts of the type "All our research indicated it's a good idea, but the feature unfortunately didn't work out and with the help of meta we could eventually improve it and now it works." I guess that is one of the possible workflows that is intended now. – Trilarion Dec 12 '19 at 11:24
  • 1
    Looks like StackExchange Inc needs a reflection similar to what Wikimedia Foundation had a while ago: mediawiki.org/wiki/Technical_Collaboration_Guidance/Vision (admittedly without any big visible result, but you can't succeed if you don't even try). – Nemo Jan 22 at 13:30
40

Thanks for sharing your thoughts with us. I can understand that this is a big departure from our past approach and may exacerbate some of the trust issues that folks here are feeling. I’d like to provide some additional color to this; specifically, how we gather feedback from our most engaged users and why Meta isn't our primary method for doing so.

We do regularly talk to our most engaged users via other research methods (for example, depending on the project, we’ll talk to moderators in 1:1 interviews, target high-rep folks in surveys, etc.). It’s important for us to be hearing from folks most invested in the site, and - especially for high impact projects - we get these perspectives early in the product development process.

The reason that Meta isn't our primary method for talking to our most engaged users is that it's not representative of our most engaged users. For example, only about 10% of our most active answerers are on Meta. So if, for example, we want to hear from answerers, we'll target answerers via other methods rather than put out a general call on Meta and hope that active answerers will respond. Other methods allow us to segment and, more generally, conduct research in a methodologically-sound way that's been vetted by our resident trained researchers.

| improve this answer | |
  • 42
    Why not do both? I understand gathering information from other sources - Meta is not and has never been representative of everybody - but I don't see why they can't work in tandem. If you use Meta and other methods, that would give the widest possible range of respondents from different perspectives, no? – Mithical Dec 11 '19 at 18:35
  • 42
    I'm upvoting this answer as it provides an honest look instead of a canned response or non-response, but isn't it possible to also gather feedback on Meta in addition to those methods? Just because only 10% of a site's population is active on meta doesn't mean that the opinions of those 10% aren't relevant, or that they are the complete opposite of the opinions of the other 90%. – Sonic the Masked Werehog Dec 11 '19 at 18:35
  • 63
    I have never once been interviewed 1:1 or targeted with a survey. I am, however, quite active on Meta, which is where I'd prefer to have these types of discussions: out in the open. As such, given that my investment with the site is self-evident, either you do not care about my opinions, or your methodology is not as sound as you appear to believe. – Cody Gray Dec 11 '19 at 18:38
  • 33
    How do you define who an engaged user is? If you're telling us that we, who spend time here on Meta, trying to help improve the sites, are not engaged users, you're going to be sending a message that is probably not intended. – fbueckert Dec 11 '19 at 18:43
  • 21
    @SonictheReinstateMonica-hog Re: Gathering feedback from Meta and other sources — this would be possible if our team was bigger. Right now because our team is so small and we have such a broad mandate, we have to be super targeted about what approaches we use and why. – Donna Dec 11 '19 at 18:57
  • 56
    In any case, Donna, it's appreciated that you took the time to answer here. Thank you. – Mithical Dec 11 '19 at 19:00
  • 15
    @flueckert If you're telling us that we, who spend time here on Meta, trying to help improve the sites, are not engaged users To be fair that's not what this A says. It says Meta regulars are a subset of engaged user, and implies that that subset is not representative of the whole. – chris neilsen Dec 11 '19 at 19:01
  • 9
    @Donna How much additional time/resources do you think it would take to occasionally pop in and look at the top answers after you solicit meta feedback? – user245382 Dec 11 '19 at 19:02
  • 9
    @CodyGray Oddly enough - and I was very surprised - I have, this past fall. I guess an email was sent out to n people who completed a public survey during the summer. The fastest to respond were "in", and it was a fairly long-ish "telephone" chat that targeted the new close messages. So I suppose it's a bit of a lottery, but that will give a better view of "everyone" than just listening to the "same folks" in Meta? And, as a mod, haven't you been involved in feedback processes the rest of us have not, recently? – Cindy Meister Dec 11 '19 at 19:04
  • 6
    @House-'ReinstateMonica'-man It's not said that the top answers hold all the important info. And trying to sift through everything that goes through Meta? With all the off-topic chaff that drifts in? I can sort of understand that that is not optimal. – Cindy Meister Dec 11 '19 at 19:07
  • 11
    @Donna It is a little naive to assume that answerers who are not active on Meta, are willing to participate in a surveys that are detailed enough to obtain useful feedback, unless you are offering cash. I generally click away any requests for a survey, especially when asked for personal information I don't deem to be relevant. – Alex Dec 11 '19 at 21:45
  • 17
    One of the big advantages of Meta over 1:1 is that it's an open discussion. Very often I'll have one initial view on a subject, read and consider points here I hadn't thought of, and realise my initial thought was wrong or incomplete. I'd inevitably give worse, less useful, less nuanced answers in 1:1 than I would here because I won't have seen other perspectives. – user56reinstatemonica8 Dec 11 '19 at 22:29
  • 8
    Meta is much more like asking your fellow SE employees for feedback than polling the community. – Mad Scientist Dec 11 '19 at 23:15
  • 8
    I used to figure that Meta wasn't representative, and is instead like a democracy -- i.e. that the community on Meta are the avid subset of the population who show up on Meta to debate and vote there. And maybe that's still there, for governance and setting per-site policies. But maybe the community on Meta are like power users - so if SE want to make the site easier and for new people -- questions about UX (like "affordances"), and some social issues (like feeling "welcomed") -- then the avid users aren't the most relevant populations to be polling. – ChrisW Dec 12 '19 at 0:05
  • 7
    As a top 0.3% answerer on Stack Overflow, I'd say that a lack of visible participation on meta is not the same thing as a lack of interest. It just means that I think there is nothing I can add to the discussion because others made the points already I would make, and possibly did so even better than I could. I'm not sure what you expect? – fuz Dec 12 '19 at 19:38
14

Online forum discussion is a valid and frequently used source of qualitative data. It has limitations, but can readily be integrated with other methods of data collection. If SE doesn't have anyone on its team that has training in these methods, it may benefit them to get training or explore the literature. There are obvious benefits to community engagement.

Meta users, and, specifically, those users who respond to a specific question, are not a probability sample, but none of the methods described above provide a probability sample either. By integrating meta discussion with other methods, SE would improve coverage as well as community engagement.

In other words, I don't believe there is a good reason from a research methodology perspective to not include "meta feedback" with other data collection methods during the research process. There may be other reasons.

Edit: based on some comments in chat, there seems to be some confusion on my meaning. I appreciate Donna's answer and upvoted it. I'm sure she's expressing her honest belief. I just disagree with her. If she saw my comment before it was deleted, I hope she found my statement of both appreciation and disagreement encouraging and helpful.

Regarding other reasons:

Another staff member with a broader role has stated that meta causes psychological damage. Even if Donna agreed with me that collecting data from meta discussion earlier in the process would be beneficial, I don't think SE would do that. Because of this, I don't think Donna's reason for not collecting data from meta is the reason for this larger business decision, the pivot away from meta.

| improve this answer | |
  • What do you mean by "above"? Only the question? References to relative positions of answers are not reliable as they depend on the view (votes/oldest/active) and changing of the accepted answer and change over time (for votes, active, and accepted state). – P.Mort. - forgot Clay Shirky_q Dec 11 '19 at 20:18
  • @PeterMortensen yes, in the question. The big screenshot and the associated paragraph where it describes the research process. – De Novo Dec 11 '19 at 20:20
  • 1
    Your answer reads as if they had never used meta feedback, so you are recommending it. But of course tgat isn't so. They previously used meta feedback, but it was unsatisfactory (to them). – Raedwald Dec 12 '19 at 0:57
  • 5
    @Raedwald I don't think they have ever used meta feedback as part of a rigorous method of data collection and analysis – De Novo Dec 12 '19 at 2:39
4

I'm pleased to say that it looks like SE has taken a step in this direction again. Quoting from New Feedback Tag – [product-discovery]:

We recognize feedback from our meta communities is invaluable and we want to share ideas with you at earlier stages to incorporate your feedback. To avoid confusion and manage expectations, we will do our best to be clear about the type of feedback we’re looking for at each milestone/stage. For example, a feature in the Discovery stage will go through several design iterations before we land on a solution, so we have a lot more room to incorporate feedback before going into Build. Whereas changes to a feature that is at the end stage of Build is costly but may be in need of user acceptance testing.

To clarify what stage a project is in and set the right expectations with you, we plan on introducing new tags (likely 2-3 overall). We’re starting with the mod-only tag on all Meta sites. This tag will be used by staff when sharing concepts in the Discovery phase relating to product or configuration changes. In most cases a direction and/or goal has been established, and there has likely been some amount of time invested in research and discovery.

This looks like it's a step in the right direction; they've publicly said that Meta feedback is important to gather before release and are actively working on ways to make that work. It's starting to look like Meta isn't being completely ignored any more and that staff has learned that they need our Meta feedback. So, kudos for that.

| improve this answer | |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .