What is health1?
I think that a healthy community... Actually, I think this is a terrible place to start. I'm in fact rather skeptical that we ought to have a universal notion of "social health" to measure internet communities, or in fact, for any community formed by free association at all, and by extension, human individuals. In short, while some sense of sustainability is usually desirable, I believe communities ought to have a right to "be themselves", if, given the surrounding social context, their general behaviour is still "legal", even though it may be odd at times.
Furthermore, I believe this right of self-determination for communities takes precedence over sustainability at all times. (if a community wishes to disband, or wishes to take actions while understanding that this will inevitably lead to disbanding, please, let them) Of course, this is likely to be against the interests of SO Inc. This is understandable, and part of my belief is that health indicators in the end do not mainly benefit the communities that are assessed, but rather the organization that derives its legitimacy by managing them (whether the organization has ulterior goals beyond managing the communities, or not). I think social health indicators mostly measure the compatibility of the community with the instance that manages them, rather than an intrinsic property of the community.
So, consider this a frame challenge. To explain my position, I will give a few examples of situations of communities below. Here, some social health indicators are rather poor, yet IMO only indicate that the community is odd, rather than bad. Sometimes, communities "just work", even if at first glance they break all the "rules". Poor social health metrics mostly measure lack of understanding on behalf of the managing organization (the community itself may also lack understanding). As such, trying to improve a community in response to poor social health metrics risks removing Chesterton's fence.
My sports club
I'm involved in a small, local (amateur) sports club. We have about 20 members. This club itself is a member of the national association for that particular sport. The main benefit of this membership for the club is the ability to participate in competitions against other clubs. Now, before we continue, please take a minute and try to think of a few indicators for the health of a local sports club. Writing it down is even better. Done? Didn't bother? In any case, let's continue.
I'm not going to guess what you've come up with, but here are some that came to my mind
- Members: the total number of members, the number of members over time, the number of new members, the demography of members (mostly age), and the sport capability of the members
- Sport achievements: the position of the club in the national competition, the individual achievements of its members, trophies won
- Promotion of the sport: publications in local news, maintaining a website, holding events for non-members
- Organizational health: responsible board and daily governance, active member participation in governance, healthy finances, cooperation with the national association, legal compliance, sensible bylaws
All these metrics seem reasonable to me. However, when I apply these metrics to my sports club, things look rather bleak. We have only a few members, and do not attract many new members. We play in the lowest division, and then at times barely scrape by. We occasionally organize an open event, but do not get many visitors. We have a board that seems responsible, but most of the few members are completely uninterested in governance. Yet, I still would mostly disagree with the judgement that my club is "unhealthy".
So, what's going on? Am I confused? Is the bias for my club preventing me from making a rational judgment? I think not. Rather, I think I forgot something when enumerating criteria. I was mostly looking at the performance of the sports club from the outside. I was answering the question "What value could this community provide for me, gazing at the community from a distance?", rather than "Why am I a member of this community?".
The latter question is harder to answer, especially when you are concerned with the good of the entire community, and hence all members ought to be considered (or at least, more than one). I am also simplifying matters by ignoring potential members, but here we still have to be careful about not looking at the community from the outside: be careful not to answer the question "Why do potential members think they would be a member of this community?" when trying to answer "Why would potential members be a member of the community?".
Besides the value implications of the perspective of the community "from the inside", it is at times also a practical perspective for achieving results that are valuable "from the outside". For example, attendance at the yearly general members meeting of my sports club was too low. We can shake our heads and scoff at the members for neglecting their responsibilities in self-governance, but that won't solve our problem.
Instead, the chair of the board asked the members that did not attend why they did not, and what could be done to make them show up next time. It turned out that most of them were not interested in attending a meeting where they have nothing to say, because they are satisfied with the current state of affairs. A few of them noted that they would be motivated to come, if we would do our sport afterwards. So, the board announced that we would have a fixed length on the meeting, and have a game afterwards. Attendance went from ~20% to ~90%. (This type of 'using selfish motivation for community goals' should not be a surprising trick. SE does this all the time. Nearly any type of desirable but tedious behaviour that is 'for the good of the community' is recognized with reputation, badges, or other forms of internet points)
Computer Science @ SE
That's enough talk about a tiny community relevant to none of you (statistically speaking). Let's pick something closer to home, and take a look at the Computer Science site2 on the Stack Exchange network, where I am currently a community (diamond) moderator. In terms of size/traffic, CS.SE is in the middle of the pack, with 8.4K visits and 12 new questions each day in the last two weeks (according to this list).
When analyzing CS.SE, there is an important aspect that didn't show up in the analysis of the sports club. This is aspect is the mission the community has beyond satisfying the needs of its members. (while this aspect is also meaningfully present in the sports club community, I talked enough about that) Note that I'm more interested in the mission as envisioned by its members (for those who have a mission in their participation in the community), rather than the one formulated or imposed by its leaders (or others). I'm not too clear about the mission of CS.SE myself, but my thoughts about it are perhaps best summarized in this recent meta answer of mine regarding the need for CS.SE, given that Stack Overflow (the site) exists. While it is perfectly fine if there are multiple envisioned missions, it is useful when there is broad agreement on a few important ones.
I think it is important that the needs of the members of a community are well aligned with its mission. When they are not, there is the risk of disruptive behaviour. One example of such a misalignment here on SE is gaining reputation for answering a question that should be closed. Ideally, members work towards the mission of the community by their selfish behaviour. This answer by Yuval Filmus, who currently has the most reputation on CS.SE, offers an interesting perspective on mission. The question assumes the mission of CS.SE is to answer any on-topic question that is asked on the site. Yuval disagrees not only with that mission, but with the assumption that he is here because of some mission. He is here to answer interesting questions3. Yet both the desire for interesting questions and the willingness to answer them fit perfectly with the result of creating a repository of high quality question and answer pairs. And judging from the voting and other answers, other members of the community roughly agree with his perspective.
So, what do the metrics says about CS.SE? The question and answering part seems to be going smoothly. There are enough questions, but not so many that there is no time to answer the answerable ones. (unlike more subjective or more practical sites, there are very good questions on CS.SE that are still very hard to answer) Question quality is good enough. There's a lot of garbage (off-topic, homework dumps, incomprehensible), but there remain enough good ones. Answer quality is also fine. There aren't many disruptions or "drama".
Other metrics are less rosy. Meta participation is low for a site of this size. Voting levels are also low, to the point where it seems potential curators lack the reputation to effectively do so. The close votes queue has permanent backlog unless a mod cleans it up. (which can be done in half an hour or so, this isn't SO) Chat is a wasteland.
As above, I'm not so sure if these poor metrics point at a real problem that requires intervention4. Sure, it would be nice if there were more curation/moderation activity. But is it nessecary? At the present, it seems not. Is this situation sustainable? Well, other than a minor exodus after the events of 2019, it seems to have been more or less stable since I've become a moderator ~2.5 years ago. How resilient is the community to external disruptions? Perhaps a bit less than others, but I think we're fine.
Could you please just answer the question?!
In short, no, apart from the bit above where I analyzed the behaviour of communities within a different framework than "health". I still really think the question asked is the wrong question, and that the term "health" obscures more than it reveals. I feel "health" conflates too many different aspects of the community and the desires of different stakeholders to be a useful concept. Still, there are two questions related to "health" that reasonable to ask.
One is about the "power" of a community. What can this group achieve that its individuals cannot? This can both be expressed in terms of its mission, as well as how it provides for its members. Completely powerless communities would be "unhealthy" indeed, in the sense that there is nothing to be gained from being a member.
The other is about "resilience". How well can this community survive and adapt to changing circumstances? A community without a shred of resilience is a dead community, so again very unhealthy indeed.
I won't spend more words on applying these questions to any specific examples, because I believe I already did. I do think that perhaps for SE sites, we can consider the "core" question and answer activity to relate to the "power" of the community, and the supporting curation and moderation activity to the "resilience". But please note that the fact that a metric exists does not mean that you should optimize for it. There may be more important goals than resilience and power, and the amount of power and resilience required to become "healthy" may depend on the matter at hand.
2: These... "entities" are often called "communities", but it remains unclear to me what exactly this means, and to which group of people it refers. (if it even refers to a group of people) As such, I will refrain from using this term in a way I do not understand, as I did before.
3: As am I, in fact. Of course, I also do other stuff due to being a moderator and all. Why am I a moderator, then? Well, someone's got to do it, and I don't seem to mind.
4: I have I mentioned the potential issues of CS.SE in different contexts before, mostly with the assumption that something ought to be done. So it seems that I have changed my mind. I think things are perfectly fine. Not "cartoon dog in burning house" "fine", but actual "fine".