53

In October we announced the beginning of our third Community-a-thon event. The event took place between October 19 and November 8, 2022.

The goals for the event are similar to those from the previous two years:

  1. Improve Empathy between all Stack Overflow employees and our sites and communities.

  2. Achieve a high degree of participation during the event, that can hopefully carry on afterwards as well.

  3. Increase our collective familiarity with our core product and elicit useful feedback on actionable ways to improve our functionality and user experience.

I’d like to report to the community on some of the stats from the event, as well as general findings and reactions from participants.

Participation & Content:

  • Out of 500 eligible participants (full time staff as of the beginning of the event who were not on leave) we had a total participation of 428 or 85.6%.

  • This included 246 staff who signed up to participate as well as 182 staff members who did not sign up but were active on the network during the course of the event and scored points in the contest. It is worth noting that after a year of tremendous growth at the company, we had more signups for the event this year than we had eligible staff for the event in 2021.

  • Participants were drawn from every department in the company.

  • Participants who signed up were asked at the beginning of this event to rate their level of experience on a 1-5 scale. 80% of users rated themselves between 1-3 (less experienced) and 20% rated themselves as 4-5 (more experienced).

To help encourage participation and make the event more fun, we had a contest throughout, where participants received points based on activity and engagement on different sites in the network, as well as for posting well-received questions and answers. During the course of the event:

  • 112 different staff members posted 829 questions and answers on 82 unique main and meta sites.

  • 29 users earned a “repeat content bonus” (4 different well-received posts on an individual site), the top finisher on the leaderboard earned it on 9 distinct sites, and the second-place finished earned it on 7 sites.

  • 128 participants earned t-shirts (or opted in for charitable donations in lieu of receiving t-shirts) by scoring over 30 points in the contest

Depiction of the t-shirt awarded to community-a-thon participants

Feedback

We ran a survey of participants after the event, in which participants were asked about their impressions of the network as well as general feedback. We got 56 responses to this survey.

When asked about their plans for future participation on the network compared to their previous levels of participation, 62.5% of respondents stated that they intended to participate more frequently, 34% intended to participate on the same level, and 3.5% were planning on decreasing their levels of participation. (In the chart below, green indicates users who intended to use the network more after the event than before, and red indicates users who intended to use the network less after the network than before).

Table showing before and after numbers from participants about intentions for network participation

When asked to rate the priority to which we should give attention to improving user experience on the network for users, the following were all rated as being important (from most important to less important):

  1. New user onboarding
  2. Asking experience
  3. Content discovery
  4. Answering experience
  5. Subjective content
  6. Commenting/chat

During the course of the event, participants submitted 82 pieces of feedback on all aspects of the product and user experience (we used a question on our internal Stack Overflow for Teams instance to collect this feedback). We also received varied feedback from staff relating to many aspects of the network.

Some selections:

Event:

This really helped put into perspective what it's like to be a new user on a Stack site, which is invaluable to me in my role here! And I learned a thing or two!

I was happy that I could actually answer questions and be helpful to others in certain communities!

Onboarding challenges:

I found the attitude, particularly on StackOverflow (in the tags I felt most comfortable operating in) to be somewhat harsh. New users brought their questions to the site and were very quickly barraged with comments, which weren't particularly gentle, telling them they had asked their question poorly with little explanation on how to do better. Sometimes a link was provided, but no actual encouragement. Often these questions were subsequently closed by votes within the first few hours of being asked. The users never seemed to come back to try and improve their questions.

As a new user, it's very difficult to gain enough rep to participate, especially if you're more interested in/able to do curation than in answering questions. The constantly full edit queue was a major roadblock.

There's a really steep curve from joining to being able to participate, and felt like a lot of trying to click things to learn that I didn't have the rep to do them (like upvoting when I didn't have the rep to do so).

Advice for new users:

I thought the new user experience was great. For someone else just starting out, I would highly encourage them to view the help section for whatever network site they are about to post on so they can better understand the moderation rules and have a much lower chance of getting their question closed.

Draft your questions THOROUGHLY before posting

Don't expect to earn significant rep from edits or upvotes at the start. The climb is long and slow.

Barriers to participation:

Most of the questions I can think to ask are already on the site. As a new user, I can't even upvote these questions or answers which means I have no way of interacting. The questions I can answer usually already have a pretty good response. As a new user, I also can't comment on answers that I think could use improvement. Without enough novel information, it feels crappy to create a new answer with the same information, just slightly tweaked. In the end, most of my points come from visiting sites daily, checking for questions and then answering fast enough to beat out the more active users.

Some users with decent rep just like to complain. Especially if your answer is now incorrect due to additional information provided by the OP in an edit. Something to identify that a response occurred prior to an edit and might be stale would be helpful.

I have to admit that I was surprised a bit negatively by the community as members sometimes downvote good questions (either from me or others) without giving any explanation as to why they did downvote. I would suggest linking downvoting up to a certain threshold with a multiple-choice field at least so the op can learn and adjust.

General reactions to the network:

The depth and scope of the sites was eye-opening, even though I went into it knowing that we had expansive topics in the Stack Exchange sites

I would recommend a heavier focus on meta participation. That seems to be where a lot of feedback occurs and it would be good for our staff to see that direct feedback.

My questions got answered very quickly. I found it very helpful to get an answer and not have to wait a while for the community to answer my question.

I really appreciate how fast answers are given and how detailed many of them are. This especially amazes me since people do this for free. It really shows how dedicated the community is to knowledge transfer.

I liked seeing the mods commenting on questions trying to help new users improve their questions. It wasn't necessarily clear to me when I should be commenting vs supplying an answer.

I found that each exchange was really different in standards. Some sites like Physical Fitness SE were pretty lax on questions they allowed that many times were opinions. While other ones felt very pedantic such as Chemistry SE. I think it's important to probably just view so many questions ahead of time to even get a gauge of what's allowed and what's not. The help sections could help, but we probably want people to jump in and ask good questions right away, but that means that reading some long detailed help system is in tension with that.

In summation

This is just a selection of the hundreds of pieces of feedback that we received from different channels. I am deliberately sharing both positive and negative feedback here to make it clear that we are getting a wide selection of this in all areas. Feedback is being shared and dispersed appropriately throughout the product organization as well, where it will hopefully be able to impact future roadmap and product decisions.

Looking ahead, there was a tremendous amount of positive feedback to the event itself, and we are evaluating plans for future iterations.

The event organizers are happy to respond to questions or comments left as answers below.

11
  • 16
    "The constantly full edit queue was a major roadblock.": can you explain this (second entry in the "Onboarding challenges" section)? How would the fact that an edit queue is full affect a new user? I am guessing this is about SO since very few other sites would ever get a full queue or even know what one looks like, so what happens if the queue is full? Does that mean no new edits to any post can be submitted? Or was the user maybe referring to pending edits on a specific post which block other edits from being suggested?
    – terdon
    Jan 10, 2023 at 17:28
  • 16
    @terdon It's related to SO, yes. Right now (and for the last few months at least) the suggested edit review queue on SO has been full most of the time. In the case that the queue is full, low-rep users can not submit additional edit suggestions.
    – Catija
    Jan 10, 2023 at 21:21
  • 6
    @terdon it's been filling up for years, I had issues with the same problem in 2020
    – Joundill
    Jan 11, 2023 at 1:16
  • 1
    If approved suggested edits didn't award rep, and/or rep was more easily attainable through other means, or even if the privileges people want were more readily accessible through means not related to filling the suggested edit queue, maybe the queue wouldn't always be full. Has there been any investigation into how... useful suggested edits actually are? Are they really more useful than pointing users to proper dupe targets or closing posts that are unclear, too broad, or otherwise don't fit within our guidelines?
    – Kevin B
    Jan 11, 2023 at 16:32
  • 4
    All of the above is perfect to go in an answer, not in comments here. Thanks! Jan 11, 2023 at 18:48
  • 4
    Re: the edit queue, I was probably one of the squeakier wheels about this internally, and talked about it at length back when the event concluded.
    – Aaron Bertrand Staff
    Jan 11, 2023 at 23:50
  • 2
    @AaronBertrand It's all down to one bad change back in 2017, before which there was basically no such issue. I brought it up on MSO a while back and re-raised it in an answer here. Jan 12, 2023 at 5:55
  • 3
    I can't say enough good things about the community-a-thon initiative, it's clear staff as a whole gained a lot and it will help y'all make the site even better. The bottom-line: enjoy your participation as regular users, that's the core experience this network is built on.
    – bad_coder
    Jan 17, 2023 at 17:03
  • 1
    @bad_coder "the core experience this network is built on" Just want to add that reading the feedback from staff above reminded me of the core feedback the user community has given over the years, which on the one hand seems to show that the problems are really there and on the other that there are still unsolved problems. What is maybe newish is the "there is nothing simple to answer anymore" vibe. In this case I always suggest to try to improve existing answers instead even though that gives much less rep comparably. Jan 23, 2023 at 7:28
  • 1
    @Trilarion the network is actually built on a "need" to get quality Q&A, but if the experience isn't enjoyable it'll be unlikely to retain experienced users or renew the userbase.
    – bad_coder
    Jan 23, 2023 at 7:40
  • 1
    @bad_coder Just wanted to say that the feedback is familiar. Of course there will be new questions from time to time but if the fun is only in building up a completely new knowledge database from the beginning and answering the same old questions just another time, then maybe the fun is over?? Jan 23, 2023 at 7:42

6 Answers 6

29

The first response to barriers to participation is pretty interesting. As a user, I'm happy when someone else on Stack Overflow has already asked the question I have, and I'm especially happy when it's already got a good response.

For me, it's not a barrier to participation at all, but is instead the entire reason I'm on the site.

Barriers to participation:

Most of the questions I can think to ask are already on the site. As a new user, I can't even upvote these questions or answers which means I have no way of interacting. The questions I can answer usually already have a pretty good response. As a new user, I also can't comment on answers that I think could use improvement. Without enough novel information, it feels crappy to create a new answer with the same information, just slightly tweaked. In the end, most of my points come from visiting sites daily, checking for questions and then answering fast enough to beat out the more active users.

I think what this (and various other pieces of feedback) indicates is a difference in the way that the community and SE Inc. view the various Stack Exchange sites.

Is the question/answer or the user more important?

6
  • 14
    The goals of the 'Thon probably exacerbate this frustration, yes; the employee found it hard to earn their points in the Stack Exchange event. But this is also a problem in real life - new participants find it hard to "break in" when the low-hanging fruit has already been plucked and all you can do is try to ask new questions and hope they don't get closed as duplicates because you didn't search diligently enough.
    – tripleee
    Jan 11, 2023 at 5:50
  • 15
    This is a reflection of my own experience as a new user, rather than the community-a-thon, but I did find this to be a barrier. I was pleased to find answers to my questions without having to ask them, but wanted to give back to the community by upvoting the helpful questions & answers, which I wasn't able to do. For me, the frustrations wasn't about gaining reputation per se, but the reason I had created an account was because I wanted to be able to contribute to the community by voting.
    – Tim
    Jan 11, 2023 at 10:49
  • 3
    This, I would say, is the response of fully computer literate participants. Have a look at the VBA tag at some of the questions asked by 1 point usernames and you will see that their question may have been answered but the user would not know how to find it.
    – MT1
    Jan 11, 2023 at 16:58
  • 6
    The issue here is how one frames participation. If it is just in gaining something from the site, then yes, it should be great to find an existing answer. If it is actively engaging with the site, leaving comments or generating content, improving posts, etc., then having a lack of things to do would be problematic. I think the Community-a-thon is probably geared almost entirely toward the latter.
    – TylerH
    Jan 12, 2023 at 14:19
  • 7
    When I saw that first sentence of that feedback, my first though was "Well, that's pretty much the point of building a Q&A repository, isn't it? Especially when you've been at it for over a decade."
    – Dan Mašek
    Jan 17, 2023 at 21:42
  • @DanMašek: Yes and no. Yes the goal is to build a repository, but no the goal is NOT to slam the door in the face of newcomers. Most notably, being able to provide feedback (votes), edit typos, etc... are privileges that are locked up. Maybe newcomers who complete the tour should get 5/10/15 points and thus unlock the first few privileges so they can engage and feel a part of the community? Jan 26, 2023 at 15:03
24
+50

As a new user, it's very difficult to gain enough rep to participate, especially if you're more interested in/able to do curation than in answering questions. The constantly full edit queue was a major roadblock.

This is all thanks to a very bad change made to the top bar in 2017, to remove the always-shown count of pending suggested edits to 2k+ users. Prior to that change, it was extremely rare for the edit queue to be full (or even 10% full) at all because people were being notified regularly of pending suggested edit reviews. That's despite the queue size being 60% smaller (200 instead of 500) and that three approvals or rejections were required to review an edit (a site-specific override implemented on SO).

The team was aware at the time of the eventual outcome it would lead to, but opted instead to implement band-aid fixes (increasing the queue size to 500 and reducing the required number of reviews to two), but those only lasted for some time.

I hope this experiment involving experiencing a full edit queue firsthand will lead to my request to revert the 2017 change being implemented. As I point out in my request, back in 2014 most of the time the edit queue size hovered around a single digit - it was rare for it to even reach 10. I'd highly recommend looking into my request which has additional arguments for reverting the change (such as users being unable to earn badges) and pushing it internally for implementation (it was already tagged by a moderator, making an internal ticket for it, but if that ticket could be assigned a higher priority...).

7
  • 1
    A related but different proposal I made that could be implemented side-by-side with bringing back the top bar counter: For edit-suggestors with a streak of N approvals, only require one approval for their next suggested edit Jan 11, 2023 at 22:38
  • 1
    I doubt that even if the CEO himself will complain about full edit queue it will change anything. There's no single case in SE history where they reverted a change years after it was done, and doubt this one will be the first. Jan 12, 2023 at 16:57
  • 9
    @ShadowWizardChasingStars They reverted the change to revalue question upvotes from 10 to 5 nine years after it was implemented. Jan 12, 2023 at 22:01
  • 2
    It's not so obvious that those UI changes back then had anything to do with it. I'd look at the number of posted items for review vs the number of active reviewers. Similarly, SO moderators seem to face very similar issues with the moderator flag queue. My guess is that it's all because of the "quantity over quality" mind set, where the bar to asking a question is all the time lowered.
    – Lundin
    Jan 16, 2023 at 11:31
  • 1
    @Lundin My MSO post linked here links to another post with graphs backing up the claim. Jan 16, 2023 at 21:08
  • 1
    @SonictheAnonymousHedgehog Yes I've seen it before but it doesn't really say or prove anything unless you also present a graph over how many reviews that were created in the same time period. The difference between the number of reviews and the number of reviewers is all that matters.
    – Lundin
    Jan 17, 2023 at 7:10
  • related recent discussion at MSO
    – gnat
    Feb 17, 2023 at 16:37
18
  1. How many participating employees read the Tour page on the site(s) they participated on (Ideally read it during the Community-a-thon, but I'd settle for ever, if that's all you can track)? Some of the feedback really highlights how unfamiliar some employees are with the company they work for... or at least their product. One of the feedback posts you shared even got the name of the company/site wrong!

  2. It's great to see that over 400 employees participated, but can you provide a breakdown of how many people from each team (including the name of the team) participated? Not only do we not know who works at the company anymore since that page was removed several years ago, but we also don't really know what internal teams there are aside from the occasional hiring announcements here on MSE. A breakdown like "Ad team: 22 of 25. Core product: 47 of 50. Finance/Accounting: 7 of 10" would be interesting and probably helpful internally (it probably isn't a concern if Ad/Finance teams are unfamiliar with using a network site, but if there are core product team members who gave such feedback, it'd be important to follow up internally, for example).

  3. Given the point of the event was to increase familiarity with the core product in general, did the event have (or have you considered for a future event) a "pathway" for participation that was just observation-based? For example, clicking on questions and reading them and their answers, visiting Meta and reading ongoing threads, dropping into some chat rooms and lurking or saying hi/introducing themselves? Many people learn about the site a lot more that way than by asking/answering, and many concerns/criticisms about 'unfriendly' or 'unreasonable' behavior magically disappear once users aren't personally invested/involved in a given post or outcome.

  4. Were there any 'super users' among the employees (I'm thinking CMs, at least, but maybe some devs, SREs, or former moderators-turned-employees, too) participating who could give primers to users before they started participating, or who were designated as 'if you have a question or problem with the event, come to one of these people for an answer first'? I know you probably don't want too much instruction if you are trying to create a true 'new user' experience, but employees are rarely going to be able to truly experience that anyway, given the institutional knowledge and--in some cases--extra access they have.

7
  • 12
    Regarding #4, the community-a-thon team held a one-hour presentation about the event and a big part of that (about half) was an introduction to the network with examples of sites they can find, how to identify a site, general information about how to ask a question well and how to write a good answer... all of that stuff. I'm the one who gave the tour and the overview on how to use the site. :D In addition to that, we had a Slack channel people could use to ask questions or get more info.
    – Catija
    Jan 11, 2023 at 20:13
  • 3
    Worth noting that this what @Catija describes above was a change this year: in past years we've had folks assigned to a mentor, and this year we switched to the overview meeting with a slack channel. This change was for a couple of reasons: 1) the quality of mentorship varied and 2) with the hiring we've done, we had more new folks than mentors available, so the overview was an experiment to see if that would work when paired with a slack channel. Catija gave a really great presentation there, and the overview was rated highly in our staff survey.
    – Philippe StaffMod
    Jan 13, 2023 at 5:08
  • 1
    'super users' .... With @Catija involved I would say: You asked, we delivered.
    – rene
    Jan 13, 2023 at 9:38
  • 4
    @Catija you'd think that would be part of first day onboarding for employees.
    – OrangeDog
    Jan 16, 2023 at 10:52
  • 2
    @OrangeDog We do give new hires an overview of the network that is somewhat similar - I think Philippe does this himself. But this is a relatively recent addition to onboarding. The onboarding process has changed significantly since I joined 4.5 years ago, so there's likely people who never got such info. I think Yaakov also does an onboarding about the history of the site. Additionally, for people working closely with the public site, they get a series of 1-1s with CMs and devs who have been here for a while, each doing an intro of a specific aspect of the network.
    – Catija
    Jan 17, 2023 at 16:51
  • 1
    What I did was a more in-depth question asking/answering with added focus on how to find a site that fits your interests discussion - designed to give people more detailed info than what the big onboardings cover while also acting as a refresher for people who might not remember the whirlwind of onboardings they did a year prior. :D
    – Catija
    Jan 17, 2023 at 16:54
  • 1
    Thank you for the responses @Catija! It would also be great to get feedback on the other questions I asked above, if possible :-) I know #1 might not be available (and if not, would be a good thing to add to a list of things to track next time), but #2 and #3 probably should be easy to answer, I imagine.
    – TylerH
    Jan 17, 2023 at 17:00
11

This is more of a comment but the question asked for answers:

Feedback is being shared and dispersed appropriately throughout the product organization as well, where it will hopefully be able to impact future roadmap and product decisions.

So I'm wondering: when will the quarterly posts return, so we can see how this worked out (and so 404: Q1 2022 Community team roadmap not found can be set to )?

2
  • 8
    We're in a leadership team meeting literally right now, thinking about how, when, and what to put in these roadmaps to give enough richness of content to make sense, without burying folks in toooooo much info. More soon, I hope.
    – Philippe StaffMod
    Jan 11, 2023 at 15:52
  • 1
    @Philippe thank you for the update, it is good to hear that you are addressing this.
    – Marijn
    Jan 11, 2023 at 20:56
8

A couple of years ago, a blog post talked about ways to make Stack Overflow less harsh by reducing downvotes, etc. The Loop: Our Community Roadmap for Q3 2020 (Downvote Research)

Meanwhile, we were talking about encouraging more downvotes on Meta Stack Overflow. What can we do to encourage downvoting?

I found the attitude, particularly on StackOverflow [sic] (in the tags I felt most comfortable operating in) to be somewhat harsh. New users brought their questions to the site and were very quickly barraged with comments, which weren't particularly gentle, telling them they had asked their question poorly with little explanation on how to do better. Sometimes a link was provided, but no actual encouragement. Often these questions were subsequently closed by votes within the first few hours of being asked. The users never seemed to come back to try and improve their questions.

I understand that any online platform can be a little harsh, not just at the beginning, and new users tend to take the encounters more personally. I am all for making the onboarding process easy. But if we focus on closed and downvoted questions, we fail to see the positive other side.

  • How many new users get an answer regardless of the quality of their question?

    • It's not rare to see a low quality question or a duplicated one getting multiple answers in the first few minutes of it getting posted. Although it's not positive from moderation standpoint (if the question is not worthy of staying on the network, now we probably have to cast del-votes as well), it's a "positive" experience for the OP as a new user.
  • What makes Stack Overflow a trusted source for finding your answer?

    • As mentioned in another answer, in a different context, we would be happy to find out that our question had already been posted and answered. Suppose Stack Overflow was like other forums where everyone had posted their question regardless of the quality of their post and whether it's already been asked/answered, and there wasn't any way to figure out which answer is better. Would you still consider searching SO for an answer?! Moderation (close votes, up/down votes, etc.) is the key to maintaining the site's quality. Users are essential, but without the content that has been generated and maintained for the past (almost) 15 years, I am not sure we'd see an influx of new users!

Rereading this question of mine from 2020, I found a couple of great suggestions to improve the experience for new users while not sacrificing quality. Downvotes research: why do we need that?

3

This is more of a comment, but you did ask for separate answers so here goes.

When I read the responses so far and the comments and feedback your participants gave I am struck by the fact that much of it seems rather Stack Overflow heavy. This is no surprise since most of your people are more or less techie and it is the poster child for the whole network. May I therefore suggest that, when you run it again, you urge them to also try some of the lower traffic sites.

When I joined the network I was only on Cross Validated and didn't find any problem finding unanswered questions. Obviously they have to have the necessary skills, but there are also more lifestyle sites which they could choose. While the event was running I did spot one person who seemed to be taking part in one of the rather odd mixture of sites I am on (sorry, forgotten which one now), but only one.

And apologies if you did do this and I just have not been aware of it.

4
  • 2
    My understanding is, this is precisely what occurred. "112 different staff members posted 829 questions and answers on 82 unique main and meta sites."
    – Kevin B
    Jan 12, 2023 at 21:15
  • 3
    It might be that we had overrepresentation on the closing survey from folks who spent more time on Stack Overflow. But most activity during the event was on other sites. That said, we did ask devs to try starting out with fresh accounts on Stack Overflow and try to report on the onboarding experience, so some of the comments are related to that as well. Jan 13, 2023 at 0:02
  • 4
    What Yaakov said. But I will add that in addition to that request (that devs try out SO with fresh accounts) we STRONGLY encouraged folks from our non-dev functions to consider trying out the other network sites. Frequently, those are a better fit for new staff that aren't devs (and therefore don't have the requisite knowledge to even ask an intelligent question on SO - and I include myself in that list).
    – Philippe StaffMod
    Jan 13, 2023 at 5:04
  • 2
    @Philippe well I can only honestly report what I read into the quoted material. Clearly there is more going on than I perceived.
    – mdewey
    Jan 13, 2023 at 14:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .