9

Related:

Executive summary

  • Could the devs please make a weekly updatable dashboard with metrics outlined here shown in graphs and a table?

  • This information is vital for continued health of both graduated and beta sites, so it makes sense to implement it across the SE network.

  • Figures from the dashboard should not be relied upon in graduation decisions - the existing 10.0 QPD (Questions per day) threshold is a clear enough goal.

Ramblings

Last April Tim Post hinted at a site health dashboard being on the Kanban wall.

Yesterday abbyhairboat asked for ideas to re-invigorate community introspection after community Q&A surveys are phased outterminated with prejudice.

It looks like all SE communities need multifaceted, concise, objective, methodologically stable, ungameable measurements of site "health" to implement timely corrective actions. All too often I read meta questions "How are we doing as a site?" and all I read in response is opinions. There's some mighty disconnect between reality and perception and having a cartload of graphs and a table of fresh numbers without the hassle of going to SEDE and manually firing a one-off query will help bridge the disconnect.

Basically, moribund communities have to open their users' eyes and healthy sites have to stop worrying. I figured out some metrics and would be grateful to SE developers if they could bring the ideas to fruition instead of the rough and antiquated Area51 system which doesn't even deal with graduated sites.

For Stack Overflow, some of those indicators could be applied on a per-tag basis to account for fragmentation of SO.

The dashboard should be publicly accessible. Since it would not include the graduation QPD metric, people will be less likely to attempt gaming the system.

  • 4
    There is a lot of data that you want to gather and display. A lot of it sounds potentially useful, but I am not convinced that all of it is or that proposed thresholds are correct (for example, the "anything greater than 0 by CMs signals a disfunctional community" bit is misguided). I fear too many people would focus on numbers and numbers alone. We should be cautious about providing data for the sake of data. That aside, the goal of Abby's post was to look at things that can't be measured (e.g. subjective quality of posts). tl;dr: much more work to be done if we decide to go this route. – Adam Lear Jul 30 '15 at 0:00
  • @AnnaLear - I'm afraid I missed the point. Totally. Sorry about that, must be senior moments here. I agree with you that Stack Exchange team does not need quantitative data becoming available to the communities. In this view I have voted to close this question. – Deer Hunter Jul 30 '15 at 2:34
  • To be clear - this isn't a bad idea in principle and you did excellent work identifying interesting metrics. We're just a ways out from throwing a bunch of them on a page. We are working on making more numbers available to high-rep users, so your proposal could be an extension of that down the road. We'll have to see what effect making more numbers public has and whether those particular ones are useful. – Adam Lear Jul 30 '15 at 2:39
3

I think you're choosing far too many and too complex metrics. I'd personally be interested in many of them, but I don't think they would be all that useful in practice. The metrics we can actually measure easily often don't tell the whole story. The one big thing we can't measure is quality, and that is the most important metric.

I strongly dislike the current display of metrics for beta sites, and I think something closer to what you propose would be a large improvement. The big improvements would be to not show any hard thresholds and to show the change over time in metrics. The absolute values are often not that useful as they can vary a lot between sites, seeing the metrics change over time is far more useful.

I would use a set of simple metrics that indicate the size of the site and the active community:

  • traffic
  • number of questions and answers
  • answered questions percentage
  • number of active users
  • number of active users at important privilege levels (2k, 3k, 10k?)

The first two are simply measuring the volume of visitors and posts. This should have a generally upwards trend in healthy sites, or at least it shouldn't decline for long periods of time.

The number of active users is important to see if the active community is growing, a decline in active users is an indicator of a possibly serious problem with the site.

The last one would be useful to see how many users would be able to edit, close and delete posts after graduation. We already have the plain number of users at those rep levels, but I think knowing how many of those are still actively using or visiting the site would be helpful.

  • Aren't we already doing that in Area 51 stats? How would you define active users? – Deer Hunter Jul 30 '15 at 9:44
  • @DeerHunter The big difference is displaying the values changing over time. Growth or decline are much more important than absolute values. There is a mod-only statistic for active users, but I don't actually know what query that one uses. – Mad Scientist Jul 30 '15 at 10:14
  • Agreed on the value of seeing the trend. There are two reasons one can't use "simple" numbers: scaling and seasonal effects. Also almost agree on unobservable quality with the exception of "rich content" indicators. Think we are in agreement that a dashboard will be very helpful, even with two or three metrics at first. – Deer Hunter Jul 30 '15 at 10:20
1

When you said dashboard, this is what came to my mind; reminiscent of that SE mini-pocket-game: (hopefully I didn't miss the point)

enter image description here

Clicking it should show you a drop down with those metrics that need the most help.

E.g, 40 questions on hold (yikes); 132 reviews queued; number of edits in the last 48 hours; unanswered, zero score, first posts (may need all of the above)... HELP!

You want people to game the system, just in a good way. And they want to game it, so help them (want to) do it helpfully.

Basically it'd be lists of things that 'need attention': Show me the on-hold list, or Show me those short, 'poor answers'. – I'll fix 'em.

E.g. drop down list:

  • 47 questions on hold

  • 97 posts flagged for brevity

  • 12 in edit queue

I realize that this is basically how the review queue works, but I think the attempt should be made to make it more user assessable and obvious that community intervention can help solve these problems (or that the lack thereof is the problem).

I must admit though, that I completely ignore the 'new users first posts' list when it's at over a hundred (as does seemingly everyone else). IMO, someone who gets paid by SE can deal with all that non-sense.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .