Is Meta no longer useful for community research?

Question

The Community Design Lead at Stack Overflow has openly admitted Meta is no longer used for conducting research on new features or for asking feedback.

Does she have a valid point or has Meta really passed its sell-by date for this kind of research?

Answer 1

I don't know if the point is valid, but the company has made it clear in various published statements that they regard meta as not being representative of the whole user base, which includes in their opinion everyone interested in programming on the asking as well as reading side, but also answerers or curators. Meta in practice is skewed towards the answers providers and curators and to those who are somehow interested in this whole meta thing, even though Meta is in principal open to everyone.

To conduct unbiased research they must either control for this skew and or find other ways to conduct research.

That doesn't necessarily mean meta is dead or not useful, just that it is only one source among many. Of course they could additionally decide to ignore it completely. They are certainly free to do that and the published post reads a bit like that.

My opinion: let them get some experience with other kinds of research, maybe the loop will make everything better or maybe next year we discuss how useful the other research is, because it may suffer from its own problems. For example 1:1 interviews may suffer from low numbers.

One example of a recent case where the meta community could substantially improve a feature: Min-Reprex: a less awkward name for MCVE

Answer 2

Does she have a valid point or has Meta really passed its sell-by date for this kind of research?

Have we become so used to hearing false dichotomies that we no longer notice when both options amount to the same thing?

And how could meta prove useful for community research if it is no longer used for that purpose?

But perhaps you meant to ask whether meta could be useful if SE were still willing to use it?

It so happens that a founder of SO has written about that very thing:

It's true that gathering feedback from your community can be hard work. And 90% of the feedback you'll get will be terrible for a whole host of reasons. It's a lot easier to imagine some heroic expert swooping in and magically blessing you with the correct answer. Well, good luck with that fantasy. The only thing I've ever seen work is getting down deep and dirty in the trenches with your users, communicating with them and cultivating relationships. That's how you suss out the rare 10% of community feedback that is amazing and transformative. That's how you build a community that gives a damn about what you're doing – by caring enough to truly listen to them and making changes they care about.

Perhaps SE should take the advice of its founder.

Answer 3

I expect that Meta is avid users -- people who choose to come here, to debate and vote (maybe habitués).

Maybe, I don't know, that's still the right (or almost the only) sample, for governance issues -- e.g. for defining per-site policies, reviewing decisions of per-site moderators, drafting and perfecting the per-site topic-specific and community-specific FAQs.

Maybe that's not (or is no longer) the right sample to be polling, for some selected other topics -- e.g. for UI/UX software changes which hope to make the site easier for newcomers, for casual and occasional visitors; or even, for polls about social issues such as whether as a new or a naive user you feel welcomed.

There was a time when we were all (or even mostly all) new here -- I doubt whether that's true anymore.

This is not to say that the community-on-Meta are all passé or even déclassé, individually at least, but it's sensible to be open to input from other users too.

It's like pleasant to have people (i.e. SE) design software for you (i.e. for we-here), OTOH I'm suspicious of companies who only scratch their own itch -- it's sensible IMO (and hopefully, also, good) that the ... the product design professionals, at SE ... will be or have been engaging with a broader population of users.

I doubt that selection is perfect, and I expect there's always some sample bias when people are asked to cooperate in a survey (rather than e.g. just being observed behind the scenes), but it might be better.

Answer 4

The comparison may seem a bit contrived and not appropriate, but the question of whether meta is useful for community research has some analogies to the question whether democracy is useful, because it obviously only captures the opinions of people who vote.

Everybody who wants to contribute to the site not only in purely technical form (Q/As), but also in terms of policies, feature requests and bug reports, can go to meta and chime in. That's a good thing. People who don't care about these things are free to ignore meta. That's also fine.

But when the staff or company ignores meta, then they exactly dismiss the people who are most engaged, and who want to invest their time and efforts to make the site better, not only in terms of contents, but also in terms of structure and organization.

In that regard, it could be boiled down to the question of what "better" means exactly. The answer is somewhere between "Making this she best site for high-quality Q/As" and "Making this a place that is welcoming for everybody". (Of course this is not a strict dichotomy, but the extremes are mutually exclusive, period). The recent behavior of the company, somehow "culminating" in the perceived attempt to replace meta by some surveys (that seem to be flawed on every level of analysis) has shown what the priorities of the company are and that these priorities are vastly different than that of the most engaged user base.

So coming back to the comparison with "democracy" and the question in the title: Meta is useful for doing research about the opinions of users who voice their opinions on meta (that sounds like a truism, admittedly). But it is not useful for doing research among the "community" in the broader sense. This "broader community" may be the people who do not visit meta, or even the community that the company wants to build.

(The latter means that this community does not even exist yet, which raises some difficulties when it comes to polls, surveys, or democratic processes. And there are questions that go far beyond that. For example, whether or not it is worth trying to capture the opinion of people who consider things like Stack Overflow as some sort of "Homework-Answering-Machine". Or whether the actual opinion of the broader community isn't eventually overridden by a political agenda that is driven by the company. But this has already been discussed elsewhere...)

What has become clear, and what may be the actual, relevant answer here: Meta is no longer useful for the goals that the company wants to achieve. And that's enough of a reason to ignore or dismiss it.