How might the Staging Ground & the new Ask Wizard work on the Stack Exchange network?

Question

We have been working for some time now researching ways to improve the new user onboarding process while also addressing issues with question quality from new question askers. Based on our user research, planning, and testing, we have recently concluded an experiment on Stack Overflow testing a new wizard for asking questions and have introduced a new proposed area of the site called the Staging Ground which we plan on releasing as a limited test in a few months.

You can read more about how the Staging Ground will work in the following posts (if you have limited time and only want to read about the details of how it will work, you can get by with just the first two posts):

1. Workflow 1: Question Details & Actions

2. Workflow 2: Listings, Filters, Quality Control, and Notifications

3. Staging Ground: Reviewer Motivation, Scaling, and Open Questions

4. New User Experience: Deep Dive into our Research

5. General introduction

A summary of the section (amended from the first workflow post):

• The Staging Ground will be a new limited-access area on the site where first-time question Authors can interact with experienced users and receive guidance towards improving the quality of their questions before they go live.

• The inspiration for the name is that of a staging environment, commonly used in developer parlance to denote a limited-access area where code can be reviewed and tested before it goes public.

• The general goals of the Staging Ground will be to:

  • Address quality issues that exist with first questions by lowering the close/deletion rates and improve their overall quality

  • Take the pressure off of the First questions queue

  • Improve the question-asking experience for new users by creating a lower-pressure environment where users can get more guidance related to asking questions. Encouraging new users to iterate on their questions in a more collaborative way should improve their overall experience and increase the likelihood and quality of future contributions to the site.

• Our initial plan is to build out an MVP (Minimum Viable Product) and release it in a controlled A/B test on Stack Overflow, after which we will evaluate feedback and results, and make a determination for future iterations and whether it will graduate. Our exact test criteria will be discussed more in a post immediately before the test goes live.

• We do not yet have a date for when this test will begin, though based on planning, it will probably be some time during Summer (June–August) 2022.

• Qualifications:

  • For questions: same qualifications as the First questions review queue (the Author’s first question on the site, or their second/third question if their first question(s) didn’t do well).

  • For Reviewers (this is how we are referring now to the experienced users who will be able to review questions and provide guidance in the Staging Ground): same qualifications as review queues (at least 500 rep).

  • We haven't yet decided what percentage of eligible First question Authors and Reviewers will be included in the initial A/B test. We may start small and increase participation as the test proceeds, with the goal of trying to ensure a good experience for both new Authors and Reviewers.

• Comments and questions will be "private" (only open to those who have access to the Staging Ground as Reviewers). There will not be any votes or answering. Search engine indexing will be blocked.

• While closing (for off-topic and duplicates) and flagging will be offered, Reviewers will be encouraged to use the new structured review workflow to more efficiently assist users in improving their question when necessary.

• Questions can be published on the site from the Staging Ground right away if they are approved by Reviewers. Questions that are not approved by Reviewers will be auto-published after a defined period of inactivity, except for questions that have received a close vote or flag, or where a Reviewer has asked for major changes that were not made.

• While it will bear similarities to the First questions queue (and will of course overlap in purpose), the Staging Ground is not another review queue. It will function differently in a number of key ways (as detailed in the posts linked above).

The initial test of the new Ask Question Wizard was on Stack Overflow, as will be the initial test release for the Staging Ground. SO is the place that has the biggest volume (by far) of new askers – and along with it, a very large need for improvements in the new user onboarding experience as well as initiatives to help affect question quality from new users in a positive way.

That said, if the test is successful and these features graduate on Stack Overflow, we hope to also be able to release them on other sites around the Stack Exchange network that would like to take advantage of these new workflows for helping new question askers onboard more successfully. (It is not yet decided whether these would be enabled or disabled by default for sites on the network, though individual sites would be able to change from the default either way.)

That is where you come in. Please take a look at the posts above (especially the intro to the new Ask Question Wizard and the two workflow posts for the Staging Ground) and think about the needs of your site in this area. If what we are presenting here is relevant to your site, but there are some things or workflow modifications that are missing that would make it much more useful, please let us know in an answer below.

Important disclaimer: There may be some requests that we will have to decline, even if they are relevant for many sites – and it is unlikely that we will be able to make substantial modifications to the tools to accommodate the particulars of individual sites.

If there are feature needs or areas of extra flexibility that are shared by a number of sites, it will make it much more likely that we will be able to address these. And right now, we are at a relatively early stage in developing this section of code, so this is an opportune time to talk about the ways in which these new features can help you out.

We’d appreciate it if you could share your ideas on how these features could be applied to the Stack Exchange network in an answer below. If you have questions about the plans shared in the linked MSO posts, please post an answer on the relevant post instead. If there are general questions that are more relevant to how these will work across the network, or if you want to have a discussion about this on your own meta site and reference it here, that is also fine.

There is no absolute deadline for feedback here. That said, the sooner we get feedback, the better (especially if there are areas of consensus).

Answer 1

For the sake of completeness I'll add this question, that was often posed before on Stack Overflow, here as well.

How will all of this scale?

There are a lot of concerns on the scalability of the staging ground. If we take a look at how the review queue's scaled on SO, my hopes are quite low for this project to scale well too. For smaller sites I think this project will be very valuable, especially for sites that have a relatively low(er) volume of questions on a daily basis. At the same time on those sites the need for a project like the staging ground is less, as site members have more time to spend on guiding users in the comment section.

For those smaller sites I see this being valuable especially for the new users, as they're less likely to encounter downvotes on their first question when it is well written and a clear question that meets the sites guidelines. And less valuable for more advanced users, as it doesn't add any value for them (seeing that they are perfectly capable of guiding new users in the comment section).

On bigger sites the impact for new users would be far greater, but at the same time due to the sheer volume of new users I fear that the new users vastly outnumber the amount of volunteering members. As was perfectly discussed before in several answers on the announcements on MSO, but explained perfectly clear by Zoe, whose answer unfortunately didn't get any formal response.

Answer 2

Here are the changes this feature would need to replace the Sandbox which we already have on Code Golf, and which seems to have a somewhat similar goal:

• Allow feedback gathered in the Staging Ground to be preserved for users to read it after the question gets qualified
• Require the poster to explicitly choose to move their question out of the Staging Area, and not do it automatically
• Also allow experienced users to opt in to using the Staging Ground, because we encourage even experienced users to use the Sandbox
• Allow voting. One of the things we use the Sandbox for is not merely ensuring questions follow the rules, but trying to generally maintain quality and ensure the challenges posted stay actually interesting. "This is boring" is not a valid close reason, but it is a good downvote reason.

These are some fairly major changes to ask for, which are unlikely to be worked on, so I suspect the Sandbox won't end up being replaced for us.

However, if we look at this feature from the point of view of adding another layer of protection beyond the Sandbox, then it would still be pretty helpful, and would only need one extra feature: a big unmissable message telling users to go and use the Sandbox before posting their question. So really all we need is a site-customisable text box which lets us write in

Big Unmissable Text

Answer 3

Make the "defined period of inactivity" a per-site customizable feature

Questions can be published on the site from the Staging Ground right away if they are approved by Reviewers. Questions that are not approved by Reviewers will be auto-published after a defined period of inactivity, except for questions that have received a close vote or flag, or where a Reviewer has asked for major changes that were not made.

Some of the smaller sites have lots of eager reviewers -- others don't. As a new user, if my new questions kept getting stuck in limbo, this would be very annoying.

Sites should be able to define the timeout period to strike a balance between reviewing questions and getting questions published.

Answer 4

The Wizard could be especially useful on sites that have non-obvious rules. Skeptics for example requires questions to be about notable claims, which is a really unusual rule and which tends to trip up new users. The by far most common way to meet this requirement is to add a quote from a notable source, this part could be a step in a question wizard.

The by far biggest issues I can see with the staging ground and non-SO sites is that the scale can be very different. It's probably easier to get this to work on low-traffic sites, but you might run into different problems there. Putting all this into a big feature with a separate UI is something I don't like, and I suspect could be problematic at lower volumes. Having a completely separate area can make sense if there is always something to do there like on SO. If the staging area is empty a lot of the time, as you would expect on low-traffic sites, users might not look there as often. Integrating the staged questions into the regular lists like proposed is even more important in this case, and I think should actually happen without restrictions. So those questions would appear everywhere they usually would, and nobody would actually have to intentionally visit the staging area for this to work.

One big concern in general I have with this is that it adds a lot of complexity. It's already hard to understand for casual users how SE sites work, this adds a whole layer on top of the existing complexity.

For Skeptics in particular I could imagine a use case for the staging area. One problem we have is that if a question is not specific enough about the claim, we tend to get problematic answers. The answers are not necessarily bad, but they might answer very different versions of the question depending on the assumptions the answerer made. And fixing the question after those answers exist is very messy, if not impossible. So fixing the question before answers come in is even more important on Skeptics than on most sites.

Answer 5

Comments and questions will be "private" (only open to those who have access to the Staging Ground as Reviewers)

I'm not sure about this. I think it's useful for new users to see what other new users are doing, for examples of what not to do. Also, in principle, I also want to keep as much of the site as open and transparent as possible.

If you really want to limit full access to reviewing to experienced users, then at least allow all users to see posts if they're given a direct link (which is already the case for review queues).

Answer 6

first-time question Authors

Does that mean that every user can only ever experience the staging ground exactly once?

Maybe make it depend on how many well-received questions a user has under the belt?

Maybe make it possible for a site to force all questions through the staging ground? Useful for e.g. Code Golf.

Answer 7

One word comes to mind: "Collaboration". And then the question: How can we make sure there is going to be collaboration on a significant enough amount of posts, so that both new users and existing community members experience the staging ground as a good place to collaborate?

A basic requirement for a sandbox to work is collaboration between existing community members and the user seeking guidance. If you want the staging ground to work anything like a sandbox on meta (which usually requires 5 reputation points to unlock meta participation) and not just another first posts queue, I think there should be a mechanism that enables (almost forces) that collaboration to actually take place, and limit or exclude unregistered users.

IPS is a site that sometimes feels plagued by "unregistered users". These can't ask questions on Stack Overflow, but on most other sites they can. They drop their post, and then vanish into the sunset or lose their cookie, never to be seen again. I think the staging ground may help the site a lot, as IPS is one of those weird 'subjective sites' where everyone also seems to misunderstand the site's actual topic. But it will only work if the current users on IPS can end up actually collaborating with new users on that staging ground. And a collaboration requires two parties, which from experience doesn't usually include unregistered users.

Regular users have little to no way to check if their efforts at collaboration can be reasonably expected to bear fruit. Whether a user is registered or not hidden on their profile page. As is the 'Last seen' field, but that now doesn't have enough granularity to even guess if someone is still checking in on their post. Regular users can only see 'last seen this week' or 'more than a week ago'. All of this isn't really helpful when dealing with things like a staging ground: As a regular user, if I were to spend my time collaborating on writing good quality posts on a staging ground, I want to at least have some confirmation that the OP of a post is still likely to be around to do their part of the collaboration, and make the changes that I will spend time on to carefully formulate in comments. If the user is long gone or will (with an 80% likelihood) never return, I can probably spend my time in more fruitful ways.

Unregistered users have a habit of losing their cookies, or just dropping a post and disappearing. I know IPS gets a lot of questions by unregistered users, maybe part of that is due to the subject matter. Usually, if you don't comment on posts from unregistered users within a few hours, you might as well just close it without commenting: You won't ever see them again to clarify their question or improve their post.

If we are going to expect posts from unregistered users to sit for a certain amount of time on a staging ground, how is the system going to make sure the users helping to shape up posts in the staging ground aren't wasting their time? Is there going to be some more granular indicator of latest activity/last seen? Will comments get 'read receipts'? Can we somehow make the system work in such a way that it enables (or almost forces) users that use it to participate? Can there be a hard limit on the amount of posts from unregistered users in the staging ground at any given moment?

_{PS: I know right now we are also already 'wasting time' with having these unregistered users post directly onto the site. And that a front page with closed questions doesn't look very "welcoming", so sweeping some of this under a rug will at least 'feel' better. But a staging ground full of languishing posts, abandoned by unregistered users, with no way of knowing if your effort is going to have an effect, is also not going to "welcome" other community members to even attempt to collaborate there. Especially if one of the goals of the staging ground is to further lower the threshold for participation, if you succeed in reaching that goal, I see the staging ground becoming too noisy to hold any value over a first posts queue, fast. So while I don't have any answers, I at least wanted to put these questions out there for consideration.}

Answer 8

I don't know if it has already been addressed, but I want to highlight a thought that worries me.

How can we prevent this feature from becoming a rep-mining weapon?

I mean, reviewers (the experienced users) could probably understand the questions even when those questions are still not well written and suitable for publishing. They could help the user to fix their question or not, but in the meantime they can prepare an answer and be ready to post it at the same moment the question goes live.

Of course, when you are the first to answer with a well written post, you get an advantage in terms of visibility and you can collect upvotes even if yours is not the best answer. As bad as we can think of it, it is part of the game and it is an important part of the success of this site.

So, this seems something that can be leveraged by the "bad guys" if there is nothing to prevent it.

Answer 9

This isn't meant to be negative and as such I request people refrain from attacking me, but my initial thought is why would someone spend their time in what must surely be a very tedious activity, an unending stream of very poor questions. There's altruism, but surely not many people are that altruistic to do something which is really rather dull. Is there some tangible on-site reward for doing this, beyond being a nice person?

Answer 10

the needs of your site in this area

It would be nice (e.g. for Code Golf) to have a customisable "must include [at least] one of…" for the tags field.

Background: Code Golf uses certain tags to indicate post type, and every question must have exactly one of a mutually exclusive set of tags.

Answer 11

How will this suggestion change the key problem with SE from a new user's perspective, namely that everything about SE screams 'You're not worthy' at new users? Your suggestion seems to imply that the problem for new users is that SE is too complex. Your solution - tack on an extra layer for good measure with extra rules!?

Answer 12

tl;dr: a sandbox will only alienate new users, and closing questions is toxic; instead, make a "needs improvements" banner.

I never really understood the point of a question sandbox (except maybe perhaps on Code Golf since the "questions" are challenges that must conform to a certain spec). Really, it just delays questions that need to be asked from being asked and could theoretically lower the number of questions that actually get moved out of the sandbox and asked as "real" questions if people provide answers as comments in the sandbox. Also, what new users would even want to use the sandbox. New users want their questions live so that they can be answered.

Although Stack Exchange sites stand out among Q&A sites and forums (vs. Quora and Yahoo Answers) because of the quality of information (heck, Stack Overflow is the go-to for 99.99% of programmers _{unverified statistic}), it feels like Stack Exchange is becoming more and more concerned about being a wiki of sorts, which is making it feel less and less like an inviting Q&A site and more of a cliquey forum. Because of this, I could see fewer users going to Stack Exchange sites out of fear of being told their question is stupid (or they themselves are stupid).

Communities should focus first on answering questions first, and then revising them to be high quality. The constant focus on "asking high-quality questions" is driving away new users. Without questions being asked and answered, Stack Exchange has no traffic and thus, no relevance on the Internet. I feel like if moderators and community regulars feel that it is important to uphold a high standard of questions (which they should; that keeps pros and sophisticated users interested), they should take initiative to specifically help users edit their questions (or edit them for them), and answerers should be the ones to ask for askers to edit their questions to clarify/narrow scope.

As psmythirl puts it, telling users to "go ask it in the sandbox first, then come back" (my words) creates a "you're not worthy" (their words) atmosphere and alienates and ostracizes new/inexperienced users.

If I may add, closing or "putting on hold" questions deemed as subpar because they aren't clear doesn't help with attracting/retaining new users either. It does them a disservice because they came to a Stack Exchange site needing help/advice/etc. They tried to write a question the best they could, and now their question is locked from receiving answers because a few moderators felt that it was unclear or low-quality. (It very well could've made sense to another user who could've answered it.) Closing questions should only be reserved for very low quality questions, spam, and duplicates (after verifying that the specific question has already been asked and answered).

Perhaps instead relegating new users to a sandbox and closing "bad" questions, a "needs workshopping" tag/designation should be created that says something like "This question might need some improvements to make it better" without disabling interaction (i.e., posting new answers), and make a Needs Workshopping tab (or something) on the homepage next to Active, Hot, etc.