Feedback for The Loop, August 2020: Community-a-thon recap

Question

In this month’s “The Loop” blog post, we did a deep dive and recapped our Community-a-thon event from our Q2 roadmap.

I know that the Community has been eager to hear about how the event went. We waited until now to post so that we could finish sifting through the feedback that we received, and so that we could report on trailing 30-day engagement numbers for staff (comparing numbers for before and after the event). Here is a preview of those numbers:

• The percentage of employees who visited any main site in last 30 days went from 72.76% (before the event) to 86.84% (19.3% increase).
• The percentage of employees who had engagement on any main site in last 30 days went from 24.51% (before the event) to 36.40% (48.5% increase).
• The percentage of employees who commented, posted, reviewed or edited content on any main site in last 30 days went from 12.06% (before the event) to 19.74% (63.6% increase).

Additionally, we had a question on the pre-event and post-event surveys about people's perceptions of the network and community:

• Pre: 5% were negative, 29% were neutral and 66% were positive.
• Post (from those who responded): 3% negative, 10% neutral, and 86% positive.

Please see the blog post for more information on event stats and details.

And if you want to hear even more about what went on during the Community-a-thon, please listen to Stack Overflow Podcast 263, which is all about the Community-a-thon, featuring Yaakov Ellis & Stephanie Cantor as guests.

Competition and Points

I would like to go into more detail here on the points system that we used during the event to add a competitive aspect and make things a little more fun. There are a few types of behaviors that we were trying to motivate/encourage:

1. Consistent participation: there are some rewards that you get by doing something minimal a few times a week, with bonuses if you can do these every work day in a given week
2. Visiting and participating on lots of unique sites
3. Creating good content
4. Reward feedback and chat participation
5. Extra encouragement for new users

Points were calculated through some really long SQL queries (on non-public tables that aggregate site activity network-wide), and were publicized internally a few times per week throughout the contest.

Point rules were as follows:

• Daily Site Activity
  • 1 point for every day with a visit to a content page (not the home page or questions listing page) on SO or any site in the SE Network.
  • 5 point bonus in any week where this is achieved at least five times.
• Daily Site Engagement
  • 2 points for every day with content created on the network. Content: Question, Answer, Comment, Edit (does not include votes).
  • 10 (20 for new account) point bonus in any week where this is achieved at least three times.
• Unique Site Visit and Engagement
  • 1 point for every unique site visited during the contest (max: 25 points).
  • 2 points for every unique site with engagement (question, answer, comment, edit, review, vote) during the contest (max: 50 points).
  • Additional 2 points for every unique meta site with engagement (max: 50 points).
• Votes
  • 1 point for every upvote received on a question or answer posted during the contest (8 points max per day). 1.5 points if account was from a new user to the network (12 points max per day).
• Chat Participation
  • 2 points for every day with at least one chat entry made in any public chat room.
• Feedback / Bug Reports
  • 2 points for each piece of feedback submitted relating to the usability of a feature or piece of UX on the site, or about the event itself (max 5 per week).

Top scoring participants in each category (all users, fresh users, seasoned users, teams) got a special holographic sticker after the event. All other participants got a non-holographic version:

The top scorers exceeded 250 points over the course of the event, with a median score of 31 among all participants. Almost everyone who scored over 70 points ended up with a holographic sticker.

---

Since this was such a success, we plan on making it an annual event. We are happy to answer any questions below related to the event, and are interested in feedback from the Community about the event, points system, and additional ways in which we can encourage more interaction and empathy between Stack Overflow staff and the Community.

Answer 1

Numbers are nice, but nothing beats a good selection of examples. I'm particularly interested in

3. Creating good content

Would it be possible to showcase some of the highlights, e.g. in the form of another blog post?

Here is a list of 291 posts across the network, during the Community-a-thon, by users with a staff account on Meta Stack Exchange. So this does not include any separate accounts used for the event, which accounts for the difference with the number of posts (428) mentioned in the blog post.

Answer 2

Someone needs to check their maths:

• 72.8 to 86.8 is indeed a 19% increase
• 11.89 to 24.5 is a 106% increase (it more than doubled)
• 12.1 to 19.7 is a 63% increase

Answer 3

I noticed a fair few positively-received questions and answers from the senior leadership team across the network. It was really nice to see them actively participating!

I have a couple of questions, both general and related to the participants' feedback:

1. How many staff members continued to participate after the event concluded?
2. What are some of the most outstanding pain points described by participants?
3. If asked whether the participants enjoyed engaging with the site, how many responded with a "positive" experience?
4. The chat room that was designated to be used for the event was relatively quiet. What barriers for entry were there, beyond the reputation gate? Did participants simply prefer to talk internally among themselves?

Overall, I'm happy to see that this is going to become an annual event, and I would consider any increase in regular engagement with the network among participants a resounding success. I would also consider lasting engagement to be an even bigger success, though!

Thank you for even considering running a Community-a-thon in the first place. It shows that SE is willing to get their employees involved in the very product they maintain, which has been a problem in the past. Here's hoping for even more engagement next year!

Answer 4

...Am I the only person here who is seeing this and thinking, "meh"?

To cut to the point, what I'm not really seeing spelled out in this post or in the blog post is:

• What new places people had the most friction (often misinterpreted as hostility by others). No indication on which sites people felt like were the toughest to work with or integrate into the community.
• What new sites overall were selected, and why. Stack Exchange is a big place now and there's a lot of sites. It's no surprise that the more popular sites would be selected, but to me that's not a really interesting detail; it'd be more cool to find out that people are finding value in the niche sites. It'd be nice to know if the staff members who visited these sites were going to get some lasting value out of them as opposed to doing this on a whim.
• What staff members felt like they contributed, and what they gained from the site. I don't think I need to elaborate on this one too much, but it boils down to, did you get your question answered, or do you feel satisfied after answering a question?
• Would a staff member stay in the community even if they're not obligated to after this exercise? Why or why not?

Answer 5

• The percentage of employees who had engagement on any main site in the last 30 days went from 11.89% to 24.5% (49% increase).
• The percentage of employees who commented, posted, reviewed or edited any main site in the last 30 days went from 12.1% to 19.7% (64% increase).

(I think there's a preposition missing before "any main site" in that last bullet point.)

How is "engagement" defined here? If people can have engagement without commenting, posting, reviewing, or editing, but more than just visiting the site, then what does that engagement consist of? Flagging? Actively reading posts? Clicking vote buttons even if they don't have enough rep? I'm especially confused because in the "before" figures, less employees had "engagement" than the other things, but in the "after" figures, more employees had "engagement" than commented/posted/reviewed/edited. That must mean it's possible to do some of those things without having engagement?

Answer 6

Simply: thanks for pushing this.

It is super simple: the more SE Inc. employees are being active members of SE network communities, the better. Every single time an SE Inc. employee turns to a community, that is good for both sides.

Thing is: there isn't a perfect strategy to get a workforce to do something specific. Sure, you can ask them, you can order them. But obviously: what you want to achieve is real buy in. The employees need to come to the point where they think "I actually like what my boss wanted me to do, so I will continue doing it, simply because I like doing it".

That community-a-thon is one way to get people moving. Sure, it sounds a bit technical, and somehow feels "typical US manager style". But as said: no strategy is perfect. If you do nothing, the community will complain. If you do X, some members will complain, if you do Y, others will complain. It is what it is, isn't it.

But me personally, I am simply glad that you folks decided to do something. It was a first step, and it got some positive results. Now keep the momentum, that simple.

The point I disagree with:

Since this was such a success, we plan on making it an annual event

Definitely a less than optimal approach.

Community participation shouldn't be asked for/rewarded once a year.

Simply make "community participation" one part of the goals you set for your workforce. Better not some stupid metric based "you need to gain X reputation on Y communities". But I think it would be useful to have some soft goals, and have your team leads/managers talk about that aspect when doing their occasional 1-on-1 meetings with the people reporting to them.

Example: in our org, we do sprint review meetings every two weeks. At one point, we added the idea of spending 5 minutes on "recognition": we simply sit down and think who did something special that helped us, and then think up good ways to say "thank you". You could do something similar: set aside a few minutes in one of your "repeating" meetings and encourage your workforce to speak about their experiences as SE network users there.

Answer 7

It's great that you tried this. I'm most interested in the next steps hinted in the "Feedback" section. That sounds exciting.

Back here in the present, though, I have a question about one of the bullets:

• Confusing engagement options

Can you provide some examples for this? SE certainly has lots of "engagement options," and I'm curious what your dogfooding has surfaced in ones that are (and are not) confusing.

Answer 8

I'm going to disagree on a few points, and ask a few other questions

@nvoigt I can't give a full breakdown by department here, but I will say that a very good portion of the company is involved in Sales and Operations, departments where no engagement on sites is needed.

A good chunk of the products SE has are intrinsically (careers, ads) or extrinsically (Teams, SE Enterprise) linked to public Q&A so while not entirely necessary, it seems... useful. Depends on where you are on ops but knowing what your co-workers do day to day also seems handy.

Consistent participation: there are some rewards that you get by doing something minimal a few times a week, with bonuses if you can do these every work day in a given week

Any way to work out what retention is like internally and get feedback to why if folks drift off after 6-8 weeks?

Spevacus' answer raises a few excellent pain points

Chat discoverability is also kind of poor. And it can just be kind of intimidating to someone new.

Chat is nearly impossible to find, and that's been one of my pet peeves for a while. Its a bit of a shame really since in many of the communities that spun off from SE, its well loved, and if it had been in active development, its likely it could have benefited, and helped the company benefit from the whole remote work thing.

Chat ought to be brought up to speed to the rest of SE (themes and so on), handling some pain points (My kingdom for an API!) and shudder maybe have a few features added on.

I'd love to see some thought into integrating it with teams and enterprise to flesh out the non public offering a bit more for companies who don't want everything on slack.

when you show up, you see a bunch of people talking about stuff (and they seem to know each other), and it can be unclear how someone should join who is not used to the medium (or feels like they are not part of the clique)

Which is kinda true of most chat rooms.

Some folks just want to do Q&A, and it is unclear how chat should or can complement that.

That said, we did have fabulous feedback about our mentor/mentee pairings, and are going to try to think of ways to potentially make things like this work with chat moving forward as a way to help new users find their way around (kind of like what we did a few years back with mentoring, though we need to find a way to make it more scalable and discoverable).

Well.. that might work, though it will be interesting to balance "mentoring" with "people who want quick answers and don't want to actually ask on the site". Root Access actually has a "soft" no questions rules cause of that (soft since you can totally nerd snipe us).

Answer 9

I know that the Community has been eager to hear about how the event went

Do you mean this guy?

I am not sure why would SE users want to know about some internal event of some company. It should belong to a company blog for investors and job seekers.

How does it affect users and sites? Why was it announced and featured?