Introducing new user onboarding project

Question

Following the Public Platform Q2 2021 roadmap, the team is kicking off the new user onboarding project and announcing our plans for the product-discovery phase.

New user onboarding – Purpose

We see about 200k sign-ups a month on Stack Overflow alone. These newly registered users (signed up within the year) contribute 30% of questions and 50% of question closures are for questions asked by low rep users (< 10). Asking and answering questions can be intimidating for even the most veteran of users and different system barriers make participation on the site difficult. By providing upfront guidance about site mechanics, community expectations, and content quality standards, we can help new users become confident contributors on Stack Overflow and the Stack Exchange sites.

As new users become more successful at crafting better questions and answers and ramping up on community standards themselves, we expect less work maintaining content quality for our curator community.

Goals

To put it simply, we want to create an onboarding experience that lowers the learning curve for new users and encourages them to participate in the community. We’re not bound to any solutions right now, but we would like the experience to include the following aspects:

• Focus on teaching new users – provide all they need to know to start off on the right foot
• Be action-oriented – help users successfully complete tasks
• Deliver value at each step – each completed task should feel like an accomplishment and motivate users to take the next action
• Introduce the Stack Overflow ecosystem – expose SO users to other aspects of our site (Stack Exchange sites, Teams)

At a minimum, a robust onboarding experience will help set expectations about community standards and, at best, help new users successfully engage on our platform.

Next steps

We are in the early stages of user research. You may have seen questions related to new user onboarding in the site satisfaction survey. We will provide updates on Meta as we continue to plan out our discovery approach. If you are interested in participating in future research, please visit your email settings and opt-in to research.

As always, we are interested in the community’s thoughts and welcome discussion on this project.

Answer 1

There's a lot to applaud here, and a lot to comment on. But I'll limit my feedback to one of each...

So, first and foremost: good! Glad to see y'all tackling this. It's a thorny problem, and one that has been with these sites since the very beginning. All too often, it has been discussed as though the solutions are both simple and obvious... Of course, they are neither; else it would've been solved a long, long time ago. So I'm happy to see you giving it the care and thought it deserves.

Which brings me to my comment

Be careful how you define the problem

This jumped out at me:

We see about 200k sign-ups a month on Stack Overflow alone. These newly registered users (signed up within the year) contribute 30% of questions and 50% of question closures are for questions asked by low rep users (< 10).

It's a little bit subtle, but the definition for target group shifts twice within that second sentence: first from new signups (200k/month) to signed up within a year of asking, then from signed up within a year of asking to asked and have low reputation. That's three groups: new users (most of whom don't ask questions), new users who ask questions, and (new or old) users who have a low reputation score and ask questions.

You... May be inadvertently setting yourself up for failure: reputation is not a proxy for "new", it's a proxy for "hasn't done anything anyone else likes". The danger here is that you're looking for the former and - especially as you push forward with improvements - you end up mostly seeing the latter... Thus obscuring the effectiveness (or lack thereof) of your efforts. Here's a quick & dirty diagram:

To put this another way: folks who ask their (first, second) question and get it right will tend to not stay at < 10 rep very long. Folks who ask their (first, 2nd ... 10th) question and get it wrong every time will stay at < 10 rep longer. If you use that definition for your target group, you'll tend to find your observations skewed heavily by folks who are the most immune to your efforts, as they will stay in the group longer and their questions will comprise a larger portion of your sample.

Choose metrics that are directly linked to the goals you wish to achieve

To address Mad's comments below: I'm definitely not arguing that low reputation is a definitive indication of someone who cannot or will not ask good questions; rather, I'm suggesting that it is orthogonal to anything this project should concern itself with.

If the goal is to focus on new users, then set metrics that are directly tied to some definition of "new-ness": asked first (or second) question, provided first/2nd answer, suggested first edit, spent first hour browsing, etc. Something that can be directly measured (with some effort) and can be normalized so as to not be skewed by unexpected changes in some other metric (voting, large increase in working from home, overall traffic).

I made this mistake repeatedly, in a slightly different context: the quality-ban system. As you know, this system affects a tiny minority of users. However, they are responsible for a disproportionate number of questions. We spent quite a lot of effort on trying to improve guidance for them, looking for some change in "recidivism" - the rate at which they were banned again. This had no perceptible effect on the overall quantity of poorly-received questions however, and our efforts came at the expense of those who could've actually benefited from the attention (the majority of people asking questions).

It was a hard lesson: everyone who asks a question here wants an answer, but not everyone is here to learn; time and effort spent on teaching is more effective when directed at those who are open to it. There's no way to know who that is, starting out - but if we take it on faith that some people are here to learn, and will benefit if the opportunity is given to all... Then I believe that good will come of this effort.

Answer 2

First off: thank you for doing this. I've often lamented the fact that new users don't know how to ask good questions because the platform doesn't tell them how to do it, and I'm delighted that you're tackling this. That said, it's important to make sure that the guidance is pushing everyone (new and old alike) in the right direction. One particular thing I think it's important to be cautious of:

Make sure we're not turning Stack Overflow into a debugging helpdesk

A noticeable number of moderately experienced users demand that users show their effort so far to solve a problem. This is the wrong definition of effort: the effort needs to go into defining a single, focused, understandable problem that needs to be solved, whether that's centering a div, undoing a Git commit, or checking whether a file exists. None of these questions have attempts in them, but they're all great questions with useful answers that have helped millions of people.

In fact, quite often these are the best questions: the ones where you know what you need to do, but have no clue how to get where you need to be from where you are. These are the questions that people are going to type into their search engine and find an answer on Stack Overflow. Half-implemented attempts often result in answers that only help the asker, because they made some silly mistake in their specific attempt. Our goal is to build a knowledge base of questions and answers that help everyone, not just the asker. The obsession with insisting users post some code showing their attempts is concerning, and risks turning Stack Overflow into a debugging helpdesk. The epidemic of "What have you tried so far?" comments (because the filter doesn't block it if you add "so far") with close votes backing them risks us building up a giant pile of questions telling the OP that they made some logic error in their specific program (hey, it's got a minimal, reproducible example of their attempt!), while losing out on the questions that will help everyone at scale.

Answer 3

New users onboarding is extremely important. So far, all improvements in the area lacked one critical aspect - telling new users about quality control, rules, and more importantly, consequences of posting poor content - question and answer bans.

I hope that part of the Focus on teaching new users will include that.

Few other notes:

encourages them to participate in the community.

Based on the sheer number of new users and questions they ask, this is not a problem. New users don't need to be encouraged to participate, they are doing that already in vast numbers. What they need is better guidance before they start participating.

Introduce the Stack Overflow ecosystem – expose SO users to other aspects of our site (Stack Exchange sites, Teams)

Introducing other SE sites is good. Too many new users will land on Stack Overflow (and possibly other sites) and ask just about any question regardless of whether it is on topic for that site or not. And by on topic in this context, I mean asking general computing questions on Stack Overflow instead of Super User, asking software recommendation questions that belong on Software Recommendations, asking math questions, biology, chemistry...

But informing and educating new users about existence of different sites is one thing. Mixing content from different sites is something completely different. Please consider that last attempt, that went beyond informing, wasn't exactly successful story Technical site integration observational experiment live on Stack Overflow

When it comes to Teams, I would be surprised if anyone could miss advertisements for Teams on the new home page. But that is besides the point. New users that are coming to ask questions on public SE sites are not interested in Teams. If their organization has Teams I am sure they will be appropriately informed.

But what would new users with a zillion of questions and problems in front of them do with Teams account? Ask questions nobody would answer? Teams is the last thing on their mind.

Teams have enough visibility. I am sure that users that can benefit from using Teams will know about Teams existence anyway.

Mixing Teams with new users' onboarding process is like mixing apples and oranges. Please don't do that.

Answer 4

I'm sure I speak for many when I say I'd love to have some new, informed users helping out with site curation and contribution. I think two particular points will help motivate them (after you've informed them, of course), and you mention both of them in your goals list.

Be action-oriented – help users successfully complete tasks

Deliver value at each step – each completed task should feel like an accomplishment and motivate users to take the next action

Both of these are pretty important to making users feel involved with the site. Making sure they complete tasks correctly is important, too, though. We have badges for things like your first flagged post and, once you get some rep, your first few comments, but you only get rewarded for doing those things "right" when you hit a whopping 80 helpful flags or 10 comments with a score of 5 or more. Those thresholds are a decent ways away, especially for newer users.

I'm not proposing new badges, but it would be nice to see some positive feedback not just for exploring the site's features, but also for utilizing them correctly when they're first starting out. A very old feature request suggested changing the Citizen Patrol badge to be "First helpful flag." The Citizen Patrol badge is quite obviously meant to simply introduce the flagging process to new users, and given that, the change is unnecessary, but the core of that feature request's reasoning is sound: "Let's reward exploration, sure, but proper use of site tools as well!"

Try to explore that positive reinforcement a little bit more in your endeavors. Make those users want to be able to do more for the site, whether that be contributing positively or curating, by being sure to reward both of those accomplishments when done correctly!

Answer 5

My number one wish would be that newly registered users (at least on Stack Overflow) aren't immediately pushed towards asking a question (any question). Instead aim to improve the quality of first and second questions as major goal, even if it then takes longer for them to be created. If we get less weed, the whole curation task is much easier and will work better. We can take more time to find, answer and polish the questions that actually can and should be answered. Everybody wins.

This would include:

• much more research before asking, especially regarding duplicates
• more focused/clear questions
• minimal examples for debugging questions

and a way to achieve that could, perhaps, be:

• a much more intense first posts screening with lots of personal feedback and maybe a positive clearance step (like: this first question is in quarantine until it's deemed publishable)
• specific tutorials on how to search, write focused/clear questions, minimal examples (with an optimal length of the tutorials) that are displayed at the right time
• showcases (show exemplary good questions with explanations of why they are good) to beginners
• big warning signs (like: if you do not do X, Y, and Z, your question will probably get a negative score and you'll risk losing the privilege to ask. Take X, Y, and Z seriously!)

Answer 6

I'm very happy that this is being worked on, but there are a few things that concern me.

The listed goals seem to be a bit too much of a buzzword soup with little actionable information, although I am cautiously optimistic about this and hope that it'll be helpful. I'm all for new user onboarding, but it's vitally important to make sure we're doing it the right way. The community has been asking for a way to better explain to new users the meanings of downvotes and closures, but little has been done about that other than some slight changes in wording and how closures were displayed. It would be better if we could do more to help people understand that we're not their personal homework-solving service, and that closures are meant as feedback, not punishment. And, of course, teach people how to do some research on their own!

---

Regarding the specific goals listed and not the concept, I have some feedback:

Focus on teaching new users

I fully agree with this.

Be action-oriented

This is meaningless and just buzzwords.

Deliver value at each step

This is also rather buzzwordy. Am I correct in saying that this sounds like finding a way to gamify the Stack Exchange system to improve useful participation. If so, why not just say that?

Introduce the Stack Overflow ecosystem

This is just marketing and has nothing to do with improving the experience of new users.

---

I think it's more important to teach users how to research their problem first, and how to interact with regulars appropriately so that they can get the most out of their visit here.

Answer 7

Please not only help new users, but also help experienced users to help new users

While you are at it, it might be worth a look how the tooling for experienced users can be improved to make it easier to onboard new users.

Just one example:

Many inexperienced users post questions without MRE. Please make it easier for experienced users to ask for missing information. I have probably written hundreds of comments asking for a MRE, but this gets boring fast - very fast. Now I'm at a point were I no longer try to explain why or how they should make a MRE, but rely on the magic link to explain them. This fails more often than results in a usable MRE.

I think better tooling can help here. How about customisable text blocks which can be tailored to explain what is wrong with the code fragments and what is missing? Or maybe tag specific magic links that explains how to make a MRE for the specific programming language? There is a lot of potential for research how the tooling can be improved.

Over the last few years, SO lost a lot of experienced users. To avoid that the remaining ones get frustrated in the never ending stream of users who need a bit of help to understand how the site works. Make it easier for them to onboard new users. This will make new users feel more welcome.

Answer 8

• Focus on teaching new users – provide all they need to know to start off on the right foot

• Be action-oriented – help users successfully complete tasks

I've noticed a lot of users (including experienced ones) who don't know how to vote and accept answers. (I could point out some with dozens of questions who never accepted an answer.)

Reading some historical meta posts, there used to be an accept-rate that stopped being displayed while some posters seem invested in demonizing any comment that can be an effort to educate users on proper site usage. As usual one of the most constructive posts on this subject was by @Shog9. I do think that more popups are needed (beyond 1k rep) to try and prompt users that mostly don't vote or accept answers to their questions.

Because now the company is asking us to help educate new users (and I'm in favor of that) but aside from a few sober posts (like this answer by @MartjinPieters and this answer by @Bill the Lizard) it seems that if I take a risk on answering a question by a low-rep user or an experienced user who always ignores answers -once they've gotten the solution to their problem- we can't ask them for feedback in the comments without risking being accused of wrongdoing.

I think the company should take a fresh look at how these things went historically and at the contradictory signals that are being sent. "Help educate users", "but let's not build in popups reminding them to vote and accept answers". It's really not my problem to solve.

Answer 9

I think this kind of forgets one essential goal - retention of new users. And more importantly turning our bright-eyed, bushy-tailed, and slightly bewildered users into well-adjusted, productive members of the community.

We don't 'just' want new users to stay new users. We want them to grow into our next generation of curators and regular users.

Introduce the Stack Overflow ecosystem – expose SO users to other aspects of our site (Stack Exchange sites, Teams)

This also assumes that SO is the 'only' entry point to the network. There are folks who don't really use SO and might come in for other sites and onboarding them is just as important. And it's plausible Teams users might come into SO and other sites.

Answer 10

The ideal case would be to have more new users, and at the same time increasing the quality of posts (both questions and answers). Having these two objectives aligned is quite a challenge, and (it is my guess) without creative approaches they would tend naturally to have a negative correlation coefficient. I.e., the quality of posts (are there quantitative measures for this?) of new users tends to be (quite, according to any measure as above?) low.

A particular point impacting quality is mentioned here. I am bringing it to SE meta as per Journeyman Geek's suggestion.

---

From my experience, I think that low rep (say, <20, and particularly newcomers with rep=1) are much more akin to not providing feedback on whatever help (comments, answers) is provided. This is either by not replying at all, or by not reasonably following a thread (not posting needed info, etc.)

That has two consequences, in my view:

1. Not being able to follow-up to a solution.
2. Discouraging helping in questions from askers with low rep, based on a increased expectation of the above.

So my two questions are:

1. Do others have the same impression?
2. Is there any way to measure the distribution of "feedback rate" vs. rep, to confirm or reject this hypothesis?
   I understand a confirmation of the hyopthesis might be a double edged sword, as it may improve the overall quality of Askubuntu (drive more productive interactions, and a cleaner database of Q&A), but it might discourage others answering to low rep askers... I don't know which would prevail.
3. (Added item here) What can be done to encourage newcomers to provide feedback/follow-up? I think this is key for increasing quality and retention.