What data about meta has eluded Stack Exchange until recently?

Question

In the recent blog post there is the following paragraph (emphasis mine):

Opinions and feelings about the best way forward are extremely valid things to consider, but the devil is in the details data, too. We’ve spent a lot of time re-examining how we look at engagement on meta in terms of the ways people participate, and how that correlates to their participation on the main site. Looking at data about meta that’s eluded us until recently (we promise we’re not ever going to call it metadata), it’s clear that our focus there needs a lot of work and better resourcing.

What is this data that has eluded Stack Exchange?

Answer 1

Original Data Analysis

• In April 2019 there was an investigation performed that looked at the number of users who regularly post and vote on Meta Stack Overflow, and how answering on Stack Overflow predicted that participation.
• The results were that posting and voting on Meta is done by 200-300 users (on average), and that answering on Stack Overflow had a weak positive correlation with participating on Meta.
• The results also showed that answering on Stack Overflow had a weak positive correlation with participating on Meta: the more answers a user posts on SO, the more likely they are to be active on Meta. However, even at extremes of answering behavior like 10 answers per month, a minority (~10%) of power Stack Overflow answerers vote or post on Meta.

This is the source of the oft-quoted stat that Meta has 0.015% of Stack Overflow's active users. And from this perspective, the statistic is true. However, in retrospect, this does not present the full picture (as we see below). This data painted a picture that contributed to some of the well-intentioned internal thinking related to Meta that continued through the end of 2019.

And please do not try to figure out who is to blame for what here (things are always clearer in hindsight). Everyone who worked on this internally (and there were many people) was coming from a place of trying to make the best product and strategy decisions to benefit the largest number of users to the greatest degree possible given the limited resources of money, developers and other product team members, and time.

Asking a Different Question

Fast-forward to mid-January (nothing much interesting happened in the interim, right?). I asked a question on our internal Teams instance summarized as follows:

• I first queried some internal data sources to look at the numbers of logged-in users who were active (viewed a page other than the home page) on MSO or MSE at least once per month in 2019, and the numbers of users who had engaged (post, comment, edit, vote) on MSO or MSE at least once per month during that time. The numbers looked like this:

• I observed that there were on average 94K active users and ~6500 engaged users per month on MSO or MSE (and this doesn't include anonymous users).
• The engagement percent averaged around 7%, which is significantly lower than most other sites, but one can assume that many people go to MSO/MSE purely to lurk and read, and the lack of engagement there should not be construed as lack of being affected by what is going on there.
• I theorized that, though the relative number of SO users who visit MSE/MSO may be relatively minuscule, the group of users who is active (just visiting) on MSE/MSO is extremely active in the areas of site moderation.
• If true, this would mean that MSE/MSO have an incredibly high degree of influence among the users who contribute to the core areas of site upkeep of Stack Overflow.
• I submitted a request to the data team to ask them to look into the question of: What percentage of content moderation/curation activities on Stack Overflow are performed by accounts that are active on MSO/MSE, specifically looking at flags submitted, review tasks performed and post edits made, and checking these numbers for users who had visited MSO/MSE in the last 30 or 60 days.

Results

Our awesome data team (in this case the data work and analysis was run, checked, and double-checked by Kevin Montrose and Jason Punyon) crunched the numbers, and got back with the following:

TL;DR - a high percentage (50+%) of curation/moderation actions on Stack Overflow come from users who at least occasionally visit MSO or MSE. This is true for the entire range of time we looked at. These graphs give a sense at a glance:

To get more into it:

• 86% / 91% of flags created on SO are created by users who had visited MSO/MSE in the past 30/60 days
• 75% / 81% of review tasks on SO are processed by users who had visited MSO/MSE in the past 30/60 days
• 61% / 66% of post edits on SO are saved by users who had visited MSO/MSE in the past 30/60 days
• 44% / 50% of own-content edits and 84% / 89% of others'-content edits were made by users who had visited MSO/MSE in the past 30/60 days
• While it is still accurate to say that the conversations on Meta are dominated by a relatively small number of users (200-300), the readership of those conversations includes users who perform the majority of the curation/moderation actions on Stack Overflow.

The immediate repercussion of this is that our estimation of the "reach" of Meta (based on the earlier data analysis and a different set of assumptions) had been incorrectly low, as we assumed that viewership correlated with answering activity - which does not appear to be the case.

We also found that there had not been any sign of a change to the amount of moderation activities that were performed by this group on Stack Overflow since our series of crises with the Community starting in September 2019.

Subsequently, the same data analysis was also applied to look at the content moderation activities on SO by users who had been engaged on MSO/MSE at least once per month (the original question looked only at being active - visiting at least once per month):

The TL;DR on this was that a high percentage (30+%) of curation/moderation actions on Stack Overflow come from users who at least occasionally act on MSO or MSE. This is ~50% of the actions performed by users who read MSO or MSE, despite this being a smaller group. This is higher than would seem to be implied by the earlier research.

• 59% / 64% of flags created on SO are created by users who had engaged on MSO/MSE in the past 30/60 days
• 35% / 39% of review tasks on SO are processed by users who had engaged on MSO/MSE in the past 30/60 days
• 28% / 32% of post edits on SO are saved by users who had engaged on MSO/MSE in the past 30/60 days
• 12% / 14% of own-content edits and 50% / 57% of others'-content edits were made by users who had engaged on MSO/MSE in the past 30/60 days

An interpretation of all of this is that MSE and MSO allow us to:

• Reach the people performing a majority of our curation and moderation tasks, and those posting a sizable percentage (if not a majority) of our answers.
• Get feedback from the people mostly doing curation and moderation tasks on SO, but a sizable number (10-20%) of answers are also being provided by these people.

Which brings us full circle to the quote cited in the question above:

Looking at data about Meta that’s eluded us until recently (we promise we’re not ever going to call it metadata), it’s clear that our focus there needs a lot of work and better resourcing.

Many people at Stack are working on these issues. Hopefully this data and history will shed some light on the questions and data that is helping to guide us in this endeavor.

Answer 2

I believe the data was just beginning to be understood differently around the time I left the company. (The timeline fits at least.) For many years the company used meta sites as a way of communicating with the most engaged users. Sometime in late 2015, I noticed a different model of meta users. Instead of seeing Meta as influencers, parts of the company began promoting a funnel model of users. In the funnel model, everyone who visits a Stack Exchange site is at the top and you winnow down to paying customers at the bottom. Since we only had products related to the Stack Overflow brand, this model suggested resources promoting the Stack Exchange brand were only helping if they converted people to Stack Overflow. For a variety of reasons this never happened, so the rest of the network was essentially excluded from the funnel altogether.

Meta is another strange situation. Again, there's no direct way to funnel meta users into paying customers. It's obvious that efforts to convert occasional readers into customers are far better spent on features on the main site. The latest incarnation of the podcast looks pretty successful in this light since a relatively small banner draws many users into that marketing vehicle. The annual survey (which is largely intended to assist marketing) also demonstrates the promise of reaching out directly to passive users. Meanwhile, meta sites tended to be distractions.

In December, there was a very real risk the company would move support, bug tracking, feature suggestions and announcements to other mechanisms besides meta sites. The only thing left would be discussions and I suspect those were allowed to stay because they could be easily ignored by employees. (It should be noted that replacing these functions would be incredibly expensive. This was, I believe, well understood. In times of belt-tightening, I don't really see how this was supposed to work.) I'd pretty much checked out of the internal discussion by then (new job offer and all), but the situation looked pretty dim from my perch as a community manager. Meta was the only real way I knew to influence the community without changing features on the main site.

Right around the time I left, there was a post to our internal Q&A Team that pointed out that while Meta was only a tiny subset of active users if you look at people posting here, when you look at active users by how often they visited, it was a much larger percentage. (I don't have the numbers at hand, but it should be possible to draw them out via SEDE if you join on AccountId across schemas.) Even after removing some of the paths to meta such as Hot Meta Posts (which come to think of it sounds vaguely NSFW), active users on main are often enthusiastic readers of meta.

Legend has it the CMs were once tasked with responding to every post on every meta. ("I'll take 'Things that Don't Scale' for 1000, Alex.") When I left the company there was a real chance meta would be all but ignored by employees according to policy. Actually including Meta as part of the company's communication strategy is hugely encouraging to me as a member of the Stack Exchange community. It doesn't make the hurt of the last four months go away, but it does indicate there is a path toward rebuilding fractured relationships. While the company is far worse positioned to meet the challenges it's currently faced with, I'm confident in the ability of many of the remaining employees to refocus on community (or rather communities) and avoid distractions that divide rather than unite.

Answer 3

Data can be interesting in funny ways, like being so obvious once you discover probable correlations that previously eluded you. I'm going to give a very high-level answer because the specifics are still settling and I don't want to assert ahead of people way smarter than me being certain.

We've known that engagement models (or how we derive active and engaged users) for main sites and meta sites differ, and on some of our smaller sites they differ pretty vastly. What we didn't understand and what was harder to surface were the ways that the models differ, and how they differ at scale.

Folks that are heavily invested in Meta might not always show the same behaviors in how they use it, but what they have in common is how much caretaking work they regularly perform on the main sites. So, instead of asking "Who are the engaged users on meta?" one should probably say "Who are the engaged caretakers on the main site, and what do they have in common when it comes to meta?"

I don't think we have fully settled on models to the point where I could go look at the queries and write up some "Cliffs notes", but we didn't see a lot of folks when we look at who's really invested here until we reversed the way we were looking at it.

There are groups of a few hundred people on both MSE and MSO that essentially show the kind of engagement that we'd normally see on the main site, without much activity on the main site. And it's easy to see that as overall engagement, but doing that seriously undercuts how you value the reach that the meta sites tend to have, on the larger sites at least.

I think we can eventually be a lot more deliberate in how we describe and document it as we finish figuring it out and what influences it. There's also an uncharacteristically large (as compared to the last 10 or so years of growth year over year) uptick in behaviors that both point to more engagement and disengagement, which is explained by some recent events, but still calls for more study long-term. The data has always been there, we just weren't looking at it from the right direction.