17

Without getting into specifics, I've noticed an extreme difference in the overall experience of a number of different Stack Exchange communities of which I am a member. Likewise I have personally experienced what I felt was a decline in quality and/or culture of some SE sites that I have been a member of for years. In some cases, I've attributed these feelings to specific policy changes and/or a change of enforcement of certain rules. In others, it seemed as if small cliques of members take over the site and shift the culture experience one way or the other. Still in other sites I feel as though the community as a whole is strong and the site seems to be in a state of growth.

Now these are all just my humble opinions and likely every other user has their own set of beliefs around the current experience of SE communities but it got me wondering:

What mechanism(s) currently exist outside of specific complaints around a particular answer/question/comment/user/moderator that allows the community to evaluate the overall health of the SE site?

And when I say health here I mean:

  • Do the members feel that the site provides regular and valid answers to their questions?
  • Do the members feel that differing views are welcomed?
  • Do the members feel welcomed by the community and the environment?
  • Do the members feel the moderators are performing their jobs well?
  • Are members actively using the site or is there a decline in activity?
  • Do the members feel that the mechanisms of StackExchange are completing the goals set forth by that particular community?
  • Are the members happy with the scope and definition of the SE community?
  • Do the members have any specific things they would like to see improved?

I personally love the data driven results sheet of the annual Stack Overflow Developer Survey and I can't help but wonder if something like that couldn't be similarly implemented in other sites on a quarterly, biannual or annual basis to gauge how members feel about the communities of which they are members? This information could be used to drive discussions on the site about possible improvements and act as a reinforcement to the moderators and superusers that the site is headed in the right direction.

Obviously something like this would need to be anonymous, need to be very well thought out (so as to be impartial) and be supported by the community as a whole. I'm curious, has anything like this ever been considered?

4/5 Update

I wanted to respond the possible duplicate question. I agree there is cross-over and pragmatically (assuming this meta-survey is implemented) I would hope that this survey would evolve into the mechanism that I am envisioning. But I did want to highlight some specific perceived differences that popped out to me while I read the other question:

  • Site Scope - I might have misread it but this question seemed to focus only on this Meta site. I'm hoping for a built-in routine mechanism for all Stack Exchange sites not just one in particular. In other words, functionality that would become as standardized and normal place as Q&A, comments and profile setup.
  • Routine, In-built & Official - I think we are in sync with having a routine/repeating survey but I am hoping for something in-built to the system. I love the standard put in place for Moderator voting. Something like: "hey, the survey is open for a limited time" would be great. But whatever it is, it needs to be part of SE and blessed by the site.
  • Target Scope - I feel strongly that any survey should be open to everyone so as not to limit people's voices... new users, superusers and moderators alike should be allowed to take it (and limited to one entry per account). Meta characteristics such as frequency of use and ranking would need to be included in figures to make real sense of the data. And obviously, convincing members (especially Developers & Security Experts) that their anonymity would still be maintained would be a challenge but I have seen this done in large corporations so I do believe it is possible. Without metadata, however, groups of individuals are either left out and/or the survey could be easily abused.
  • Topic Scope - At the end of the day we have a lot of cross-over in topic but I don't want to downplay the culture angle. The satisfaction survey mentioned by Journeyman Geek below is much closer to my "Health Assessment" idea. At the end of the day new user experience and ideas for site improvements are absolutely necessary and I think in scope for both surveys. But I really want to make sure we are also capturing "culture" as well. Right now it feels like (at least to me) that you either love the site (at which point you invest the time to be a superuser or moderator) or you have to attack a specific problem example through the complain mechanism (which often doesn't address root cause or a general trend). By trying to look specifically at the overall experience of different groups of users and also comparing the results to historical experiences, I'm hoping to attack possible groupthink and give voice to members who might feel a little disenchanted or who would like to move the site in a different direction.

So in summary, I do think this might be a bit of a duplicate (honestly, I didn't see the other question before) but I feel like my idea might be separate enough to warrant additional discussion. I'll defer to the moderator/community, though, if we need to close this.

|improve this question
  • 1
    Possible duplicate of Let's Start An Annual Meta Survey! – Sonic the Anonymous Hedgehog Apr 5 '18 at 0:31
  • 1
    there used to be yearly(??) "community self-evaluations" for beta sites. They got dropped because they weren't providing the expected value. A lot of this data-focus you describe was absent or understashed in those surveys though. Of note: Code Review has a track record of finding and fixing problems on site through such a data driven approach. Other sites might similarly find the idea valuable... – Vogel612's Shadow Apr 5 '18 at 18:23
  • Do you want to add "feature request" to your tags? (I'm not actually sure if that would make any different in the large scheme of things.) – aparente001 Apr 11 '18 at 17:17
  • @aparente001 Good idea... I've added that tag. Thanks! – DanK Apr 12 '18 at 13:18
  • I once proposed something similar to this (without knowing) on a per-site meta: quantumcomputing.meta.stackexchange.com/q/350/11 – GDPR Jan 14 at 6:33
3

I'm actually under a bit of an impression that the SO Survey's are pretty heavily a marketing tool - for SO's job sites.

As such, there's little 'practical' use as far as SO's concerned in a 'formal' survey for a site, and certainly not to the same level of publicity that the SO survey's get.

The beta sites (until recently?) used to have site evaluations as part of their beta process which served many of the same purposes.

That said There's a reference to randomised surveys here

User satisfaction survey: Understanding user satisfaction levels based on quarterly user satisfaction survey is an important data point for most sites. In the past we have relied exclusively on anecdotal feedback to evaluate satisfaction. This month we will be building an automated system to randomly select users and prompt them via email to fill out a survey. This system will be tested on Teams alpha users and will be expanded to others over time.

So presumably this is the sort of thing that belongs there, and could happen anyway. Its unlikely to be as broad as the Developer survey nor is it likely to get the same sort of in depth attention I suspect, with a few hundred sites.

|improve this answer
  • I'm actually quite excited about your link to randomized satisfaction surveys. That is pretty much what I'm envisioning (only with a lot of detailed statistical analysis performed on the results and shared publicly). But I really think something like that is just as important (if not more so) for established sites as it is for Beta ones. Groupthink and stagnation is common to any organization so having mechanisms like that to combat it is IMO essential. – DanK Apr 5 '18 at 16:20
  • Marking this as the answer as it does appear some work has been done last year in line with the survey idea above. Would love to see this grown out a little more. Thanks for finding this reference, @Journeyman Geek. – DanK Apr 6 '18 at 14:02
3

Not answering the whole question, rather than adressing this part:

What mechanism(s) currently exist outside of specific complaints around a particular answer/question/comment/user/moderator that allows the community to evaluate the overall health of the SE site?

I'm just going to list here more or less or not-at-all evident "indicators" that I've been looked at, answering some points in your definition of "health".

  1. Question rate. Stats on Area51. For beta they are helping to judge how is the site doing, for graduated they can be a rough (or not rough) estimate how the visits and q/day rate changed. The current stats can be seen on http://stackexchange.com/sites. This way you can make a comparison, somewhat.

  2. Meta engagement and participation. Actually here "complaints" would be intersting thing to see, same thing about reactions to complaints.

  3. Review queues activity/inactivity xD

  4. Moderators activity and last visit time; last election/appointment.

  5. Answers per question, comments per question, for recent entries.

  6. Closing rate of recent posts. Overall "miss" of the scope by the newcomers asking questions (looks towards Lifehacks SE). Closing rate of questions by newcomers; whether there is an explanation and reaction or not. Reopen rate.

  7. Overall reaction to the new posts, views.

  8. Whether do the questions on the site reach often the hot list or not.

  9. Whether do you understand the scope of the SE site you're "inspecting".

  10. There's data explorer to gather information about decline/rise of activity, other similar things, stuff, you know. http://data.stackexchange.com

|improve this answer
  • This is great info, thanks. TBH I wasn't are of a few of these items but they do all tell part of a larger story. I think we still need a hands on user experience survey to compliment these figures but being a data nerd myself... I would absolutely want to include this data in the overall results. – DanK Apr 5 '18 at 16:22
3

You always have the option of using an external survey software and asking people on the relevant meta and in chat to fill it out. You might be able to get a moderator to add the tag on the meta post to reach a wider audience.

It's been done before.

Anime & Manga ran surveys back in 2014 and 2015 (despite being called a "Quarterly Feedback Survey", it looks like it only ran twice). From the 2014 post:

The form is powered by Google Forms, and is completely anonymous (unless of course, you'd like to tell us who you are).

Another survey I know of is this one, which was done pretty recently. That survey also used Google Forms, but its focus was much different: why do people ask off topic questions here on MSE?

There was an official MSE survey planned (based on the other MSE survey, but larger in scope) but I'm not sure when or even if it will happen.

I've also found a survey that was posted on a meta and used Google Forms like the above, but wasn't about the site itself (it was about something domain-specific, getting a certain type of data from different cars):

|improve this answer
  • These are great steps in the right direction (and I do appreciate the links) but I really think to truly capture what I'm thinking there needs to be something official and routine. Yeah, it would take work but once questions and a mechanism for survey collection was agreed to you could pretty much automate it and make it standard across all SE sites. – DanK Apr 5 '18 at 16:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .