6

The goal of international communities is to create a world in which, for any search engine query on programming, the search results will show a link to a detailed answer in our first language. The road to such a grand goal is winding and tough; there is a chance of getting lost. It would be great to have a landmark that will help us not to stray from the right path.

International site users have been proposing their metrics that can serve as the landmark to our goal. Questions on international metas: SOru, SOes, SOpt, SOja.

Main criterias

  • The fewer metrics we use the better. The best option would be if we created a single metric to track our overall progress.
  • Metrics should be interpretable. The goal of these metrics is to show in a simple way how our efforts are bringing about change: what is going well and what the community needs to pay more attention to.
  • Metrics should be universal, in a way we can use them in any Stack Exchange community.

Key metrics: Community, Quality, Quantity

  1. Community. How comfortable are the users in the community? Are they active on the site? Do they want to invite their colleagues?
  2. Quality. How useful and interesting is the knowledge that we create to site users and overall developers?
  3. Quantity. How fast is the knowledge base growing?

Summary of the proposed metrics

The most popular proposed metric in all communities is how many active users join a community opposed to those who leave (so-called "turnover"). The only difference is in how we determine an “active user”:

(Btw, please share how would you define an “active user” for this metric?)

Interestingly that we can measure our distance to the goal literally: using services that provide statistics on search engine queries (for instance) we can determine the most popular search queries on programming and compare that with what questions we have on the sites.

Other proposed metrics:

  • The number of suspended users separated by suspension reasons.
  • Number of bookmarks with a link to the site created on social bookmarking sites (for instance).
  • For how long should a person participate on a site to answer more questions than they have asked?

A lot of thanks to everyone who shared their ideas. I’m going to look at which of the proposed metrics can be implemented and in what form. If you have any ideas on this subject, please ping me in The Terminal

Please keep proposing metrics

Creating interpretable universal metrics is hard work. Metrics can and should be continuously improved. Please keep proposing your ideas about new metrics and thoughts on improving existing ones in the answers to this question and on international metas. Together we will make it!

  • I'm not entirely following yet. Is this still focused on the international SO sites or is this supposed to be a general SE metric feedback request? – Chris says Reinstate Monica Aug 19 '19 at 10:51
  • @ChristianRau Hey! It is an initiative from / for international sites only. At the same time if we find that something works well for them, I hope there will be a good chance of having that on the other SE sites. Please propose your metrics! =) – Nicolas Chabanovsky Aug 19 '19 at 10:54
  • Not really sure what your question is here. It all a bit vague to me. – Luuklag Aug 19 '19 at 13:32
  • 2
    I'm not sure that's all that matters. The intangible stuff is just as important for a healthy community. How's the community core? How do they communicate (on and off site?). The patient might have perfect BP and blood sugar, but if they're rambling about cheese... we might have an issue. So for any community, international or otherwise - how's the company planning to get a feel of the more intangible, human side of their communities? – Journeyman Geek Aug 19 '19 at 15:17
  • @JourneymanGeek If I get it right, the company can look at the human side as the number of off-topic conversations (meta and chats). Could you please tell what would you suggest? – Nicolas Chabanovsky Aug 19 '19 at 17:58
6

Something I find interesting is the median time to questions receiving their first upvoted answer. However, this can be as distracting as it is enlightening.

The reasoning is our Q&A engine sort of makes a series of promises to users that engage with it:

  • You will be able to post a question
  • You will receive at least one answer to your question
  • The answer(s) you receive will be reviewed and vetted by the community

So, how long does this normally take for a question that is on-topic and contained all of the information needed to answer? That's why it gets rather distracting the deeper you go. Some things I've observed:

  • You should filter out the top 5% of peaks. If a question was asked one year ago and got one answer which finally got an upvote a year later, the median would be greatly affected. You have to ignore these cases.
  • Tag popularity is a heavy influence and numbers will quickly try to scare you that you're failing in your lesser (niche) tags.
  • Some questions receive a lot of vetted answers very quickly but are closed because they are duplicates. There are reasons for and against factoring these into the metric.
  • Even with dropping the top 5% of questions that took the longest to answer (and only working with the remaining 95%), questions that require rare knowledge are going to drag this metric down, even in the popular tags. Keep looking for tag patterns (e.g. frameworks) where you can eliminate outliers that inordinately weigh down the metric.

This metric isn't something I'd use to derive automatic calls-to-action, which is why I haven't really pursued it on the English site (compounded with how heavy the queries can be depending on how far back you go). But it does illuminate some areas of participation and can be (part of) a lens that communities can use to better focus energy.

  • Related to your last bullet, interesting, novel questions can require extra time to research, develop, and test answers, and then time for the community to validate those answers. Some of the most interesting questions on the network didn't get good answers right away, but they still contribute to quality. I don't know how to factor that in. – Monica Cellio Aug 19 '19 at 15:22
  • @MonicaCellio I usually observe such questions, being interesting but hard to answer, get a relatively high number of up-votes, when written well, or a major revision by a high-rep user or tag expert. Unfortunately all things that are heavy to query in SEDE I think. – Luuklag Aug 19 '19 at 15:45
  • 1
    Very interesting idea of turning a promise / goal / core value into a metric. Never thought about it from this perspective. Thank you! – Nicolas Chabanovsky Aug 19 '19 at 17:49
1

There are a few principles whe would like to take into account, as Tim already pointed out:

  • Questions need to get answered
  • Answers need to be of certain quality

There are also some things we would need to correct for to arrive at a uniform metric that is applicable to all sites in the network. This mainly has to do with the size of the different communities.

Therefor I propose the following:

100 - ( % unanswered questions / average answers per question )

Where we look at questions that are:

  • NOT closed
  • At least 14 days old ( to allow off-topic questions to get closed )

For answers we need answers that have a certain quality. Within the network quality is expressed through voting. As low quality answers don't provide any value to the network we should calculate the average amount of answers solely based on answers that are either:

  • Scoring at least +1
  • Are accepted with a score of at least 0
  • Are for a question as indicated above

So for example a site having 90% of its questions answered, and having on average 1,8 "quality" answers per unclosed 14 day old question would score: 100 - ( 10 / 1,8 ) = 100 - 5,56 = 94,44

While a site with a little more unanswered questions, say 20%, but on average having 2,3 "quality" answers per question would score: 100 - ( 20 / 2,3) = 100 - 8,70 = 91,30

  • Thank you for the metrics. They look great, though a bit hard to interpret, as I see it. – Nicolas Chabanovsky Aug 22 '19 at 16:05
  • @NicolasChabanovsky you want to score as close to 100% as possible. What an acceptible score would ve is open to interpetation, as this is simply a benchmark across all sites in the network. You can achieve a perfect score by having no "unanswered questions". Having a high average of usefull answers per questions is favourable as multiplier to decrease the weight of your unanswered score. It might prove after running a benchmark that some multipliers could come in usefull to make the metric more descriminating. – Luuklag Aug 22 '19 at 19:29
-1

Since you stated the goal as

the search results will show a link to a detailed answer

it looks like we ought to focus on answers. My preferred metric is Year-Over-Year growth in the number of answers. To focus on creation rather than deletion, I use PostsWithDeleted table. Here is the SEDE query computing YOY for 2019 Q2, the latest full quarter.

  • Stack Overflow in Japanese: +2.4%
  • Stack Overflow in Spanish: -6.3%
  • Stack Overflow [in English]: -9.8%
  • Stack Overflow in Portuguese: -16.2%
  • Stack Overflow in Russian: -26.9%

Across SE network, French Language led with 52.2% growth, while Iota experienced 86.5% decline.

  • 2
    Perhaps it would be wise to take into account the amount of questions asked during that same period as well. So the ratio of answers to questions instead of the absolute difference in answers. Since if there are no new questions to answer we could still have very active community members (checking the site every day for new questions to answer) that simply can't answer as there is nothing left to answer. – Luuklag Aug 19 '19 at 14:36
  • Why did you pick these language related sites? In any case, I do believe SE, Inc tracks these numbers and sees the overall picture you do, and this forms the commercial (as opposed to ideological) side of their push for being more welcoming. They want to get more engagement, quantity, even it it comes at a loss in quality. I believe the OP here is asking from the community’s perspective, who is resisting the welcoming initiative, as they tend to favor quality over quantity. This ties into Luukag’s comment on “nothing being left to answer”. – Dan Bron Aug 19 '19 at 15:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .