14

A community is a living organism. As we measure our pulse and blood pressure, we need to be able to monitor the community's health to understand if there is something that we should pay special attention to.

We are pleased to share with you the first results of a research project about a community health indicator that we conducted in early autumn.

Please take a moment to give it a read and let us know what you think.

2
  • 8
    Where are the first results you are sharing? The post you link to seems to be announcing a research project, but I don't see any results there. Did I just miss them? I see an announcement of a meta health indicator, a definition of what a healthy meta should look like, a list of sites that will be tested, but no results.
    – terdon
    Dec 18 '20 at 17:50
  • 1
    Looks like there's no link to this question in the post.
    – LShaver
    Dec 21 '20 at 16:21
12

I think it makes sense to keep an eye on meta activity to judge the health of a site, as a complement to activity measures on the main site. What is not clear to me from the blog post is whether this is a CM instrument alone, or something the community should be able to view and use?

What's missing from the blog post are details and data. There is a rough description of the method, and a paragraph of results. But the underlying data is not available, so the community can't fully review this indicator.

One of the things that is very hard when looking at any of the activity statistics is to judge what a "normal" value should be for a healthy site. It's very hard to get a good overview over the sites, that information is never directly provided by SE. And it's also hard to get yourself on SEDE (it's possible in general from what I've seen, but cross-site queries are quite far beyond what I could do, and I'm not afraid of SQL). There should be an easy way to compare the activity indicators across all SE sites, so that you can pick sites for comparison that make sense.

One thing that you can judge reasonably well on its own is the trend itself. Declining activity over a long timeframe is concerning in any case, no matter what the absolute value is. So the trends of these indicators should be available as graphs to the community.

5
  • 1
    You raised a few interesting topics, thanks! In a perfect world where we have endless time and resources we would build at least two different indicators: one for communities and the other one for CMs. It seems to me that these two groups have slightly different focuses. For example, for users it is important how exactly a rule is defined where CMs want to make sure that a diverse group of users participate in the discussion of the rule so that the rule meets the needs of the majority. In the blog post we shared our thoughts on the CM part.
    – Nicolas Chabanovsky StaffMod
    Dec 21 '20 at 15:42
  • I think the community health indicator should help us (CMs) get a better sense of what communities need our assistance the most. To do this, as you mentioned, we need to know what "normal" means. In this research, we give most of the definitions from a sociological point of view. For example, any group of people starts when someone from the group clearly sets the group's boundaries (i.e. how “we” differ from “them”). On Stack Exchange this process begins on Area 51 when users ask their first questions and continues in an evolution of the on-topic help center page over time.
    – Nicolas Chabanovsky StaffMod
    Dec 21 '20 at 15:44
  • I agree with you that exposing more meaningful data to the users is a great step towards building better communities. We would like to have reports like “a year in moderation” be automated and available for core users one day.
    – Nicolas Chabanovsky StaffMod
    Dec 21 '20 at 15:44
  • 2
    @NicolasChabanovsky One major concern for me would be that a red/yellow/green indicator is too simplistic to be useful, there are large and fundamental differences between the sites. The Area 51 indicators for example were constantly misinterpreted by users and even moderators, I think in the end most of them were harmful and didn't help. Dec 21 '20 at 15:48
  • Yes, you are correct. An overly simplified model can do more harm than good. I hope it is not the case for this project. The goal of the community health indicator is to provide a summary (the color) of the current situation in a community. When a CM looks at a particular site they see a very detailed analysis (but again with some levels of the scale of details).
    – Nicolas Chabanovsky StaffMod
    Dec 22 '20 at 7:32
12

So, doctor... if the community's health is failing, what's the medicine?

Are there any thoughts on what to do with the numbers yet? This is apparently a research project, but the blog post doesn't say anything about next steps. How will the gathered data be used, what can we expect?

The blog post seems to suggest this data can be used to reduce social complexity, what I couldn't really figure out is: how does knowing about meta activity, which is only a small part of the whole, help with reducing the social complexity of the entire network of sites?

And finally... I'm kinda worried this may put quantity over quality. Your metrics are mostly focused on numbers, and those can be reached in very 'unhealthy' ways as well. There was a lot of discussion here last year that brought along quite a few new faces, yet I doubt anyone would use those numbers as an indication that MSE was healthy. Is there any approach/safeguard to prevent an approach to community management that's overly focused on metrics and numbers?

5
  • 7
    I'd agree that the quality of the content is far more important than the quantity.
    – Snow
    Dec 18 '20 at 17:46
  • Good questions! Using data we can learn the true essence of the processes, or we can get bogged down in unimportant facts. The first step of the research was defining how a successful meta site should look in theory and what we should do to get closer to the ideal world if something is wrong, i.e. all defined metrics are actionable and answer particular questions. For example, in the case of Stack Overflow in Russian, I think we (the community) need to have more meta discussions on more different topics. I expect that this will lead to more people sharing their views on the future of the site.
    – Nicolas Chabanovsky StaffMod
    Dec 22 '20 at 7:11
  • Let us think about a software application. As soon as programming is done and we have an executable, we run a set of unit tests to be sure that the application meets the logic requirements. I think of the community health indicators as kind of unit tests that check if everything in the community goes according to the community management logic, and if not, where we should look at. This project is just the first small step towards making community management on Stack Exchange even more proactive and data driven. =)
    – Nicolas Chabanovsky StaffMod
    Dec 22 '20 at 7:12
  • @NicolasChabanovsky Who were involved in defining what a successful meta site should look like? Having an executable with unit tests is nice, but who defines the acceptance criteria and where are the acceptance tests?
    – Tinkeringbell Mod
    Dec 22 '20 at 7:48
  • @Tinkeringbell Mostly CMs. All the results are available in the blog post. So if there is any issue with the definitions we are happy to adjust them.
    – Nicolas Chabanovsky StaffMod
    Dec 22 '20 at 8:12
7

In the post, you define a "meta community"1 to be healthy if it is both actively used and used for its intended purpose. I'd like to know how this notion of "meta health" is relevant for the associated site2, or in particular why it would be good for an SE site to invest in the health of their meta.

From the post, it seems you treat meta usage as an end in itself. I don't see it like that, for me meta is not an end by itself, but merely a means to an end. It is a tool to (among others) improve the quality of a site. Additionally, you talk about the intended usage. Whose intention do you mean? Different sites may have different intentions on how meta should be used, or different experiences in how meta has been useful for them. I understand that you want to have tools that are generally applicable to all sites, but I worry that you're going to reduce complex issues to a single number and miss important issues as a result.

For example, you note in your case study on Mathematics.SE that it is a healthy meta community based on usage metrics. However, I was surprised that you did not mention the following in your case study: there is significant disagreement on Mathematics.SE on how to deal with low quality questions. I'm not aware of a single place that has an overview of this issue, but discussions arising from the most recent moderator election questionairre should give you some idea. Whether this disagreement is good or bad is up for debate (and I don't intend to criticize Mathematics.SE here, so please forgive me if I misrepresent this issue), but this issue seems to be significant for that site and I think it is not good that this is issue is missed in an analysis about the "meta health" of a site.


1: As an aside, what community is this referring to? Shouldn't this just be the community formed by the site? "meta community" sounds like a group of people with the primary purpose of "being meta", which is the kind of behaviour that some argue can be a risk for a community if done in excess.

2: To be clear, I will use the word "site" to denote the group of people who visit and post on <sitename>.stackexchange.com and/or the associated meta an chat space. Some people may use the word "community" for this group, but people use this word in a way I don't understand, so I will refrain from using it myself.

1
  • I think the community in which people want to stay wins today. Meta is a place where users get together to discuss how to create such a community. By measuring activities on meta, we want to understand if users are setting their site for themselves. In other words, the fact that users on Mathematics.SE care about the site is the focus of the indicator, instead of the outcome of the debate. On the other hand in the past we used another tool that measures “velocity“ of a meta post. That allowed us to look at very active meta posts over the network and provide users some guidance if needed.
    – Nicolas Chabanovsky StaffMod
    Dec 22 '20 at 8:08
-1

A lot of users are active on the meta site

This needs some context. I participate on a smaller site and the community hardly uses meta. It's hard enough writing technically accurate answers (considering average research time per answer 30 minutes upwards) that taking the additional time to engage on "meta-discussions" just feels secondary.

Meta site are also fueled by users raising "issues", and the tiny community of mods adding authoritative answers. In a site where "issues" are rare and the core community is focused on the flow of content "meta-discussions" may simply not arise.

Besides, a lot of the users on smaller sites have their core activity on larger sites like Stack Overflow so there's little need or interest in repeating "meta-discussions" that have been held elsewhere before.

3
  • 10
    If most of meta discussion is being handled by the mods, then that isn't a healthy meta. If a local meta is just repeating discussions from here (MSE), then that isn't a healthy meta. If a meta is only dealing with users raising issues, that isn't a healthy meta. If the user base feels that meta is a waste of time, then that isn't a healthy meta. On a healthy meta site, the community will discuss issues specific to that site. They can discuss site scope, local norms and rules, moderation action, tagging, events, tag wikis, canonical answers etc.
    – terdon
    Dec 18 '20 at 19:38
  • @terdon those 2 enumerations purposely miss the point I raised. (That's against healthy discussion.)
    – bad_coder
    Dec 18 '20 at 19:49
  • I'm sorry but I don't understand. What point did I miss? I can assure you I didn't miss it purposefully. It just seems to me that your answer shows a misunderstanding of what meta sites are for. You have a few examples of cases where low meta participation is not a problem in your opinion but all of your examples would indicate an inactive and unhealthy meta.
    – terdon
    Dec 19 '20 at 13:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .