23

I recently received an email from "Jeff // Stack Overflow" (no relation), containing a link to a survey. Before the survey was an email address, and an invitation to send feedback to it.

… Unfortunately, I forgot to write this email address down. Hence the openness of this open letter.

TL;DR: Some questions in this survey are wrong, and don't allow frame-challenge responses. I wouldn't be able to draw correct conclusions from this survey, so I suspect Jeff might struggle. I advise Jeff to consult meta, in future, before investing time and effort into experimental design. (Since the company's on an AI kick, here's a metaphor from Eliezer Yudkowsky about “wise old wizards”.)

Attn: Jeff the Product Manager.

Hi there,

We're excited to invite you to participate in our latest round of user research! This survey will help us better understand what question quality means on Stack Overflow.

This survey boils down to scoring questions against a three-category rubric:

Category Definition
Context and Background The question provides comprehensive context, including necessary details, background research, and a Minimum Reproducible Example (if applicable) to ensure full understanding of the problem. Previous attempts to solve the problem are also documented.
Expected Outcome The question clearly states the desired outcome or goal to solve the problem.
Formatting and Readability The question is well-formatted with correct grammar and spelling, and includes properly formatted, easy-to-understand code when relevant.

There's a much more detailed rubric later in the survey, which has clearly had a lot of effort put into it. Unfortunately, it's completely wrong. You've confused our guidance for new users for some kind of "how to write an awesome question!" instructions. Really, they're more "ensure there's enough raw information to make a good question out of" instructions.

There are questions that score very highly on the rubric, yet are long-winded and unuseful, and/or (to use one of your examples) exact duplicates of existing questions. Your survey doesn't provide a way of explaining why I disagree with the rankings, so there's no way for me to give this feedback.

(The answers I have been able to give would appear to validate your rubric, because this survey does not seek to disprove that hypothesis. I'm a little concerned that you might read the survey results as supporting it. Adding an "is this a good question?" Likert might've helped: that would at least have tested how well your proxy metrics correspond to the perceived ground truth.)

Likewise, the #2 top question (by votes) on the network, How do I undo the most recent local commits in Git?, would get 2/2/5 (out of 5) on your rubric. It contains no research effort¹, nor an MCVE, nor documentation of previous attempts. The goal is stated vaguely: as you can see from the answers, many different things might be meant by the words in the question. And it's a better question for all of this. Questions are for people who don't know the answer, and want to find the correct answer.

It would've saved you a lot of time, had you asked the first question from your survey:

(Optional) As a reviewer, are there any additional categories you consider when evaluating the quality of a question that are not listed here?

on meta, before doing all this work. (Preferably main meta: you might struggle to interpret SO-specific aspects of question review until you understand the general basics.) This might get you a list such as:

  • Is the question narrow enough to be answered within the Stack Exchange format? (yes = good)
  • Is it broad enough for answers to be generally-useful? (yes = good)
  • Does the title describe the general case of a problem while the question body asks about a specific case? (no = good) If so, I'll usually edit to fix that.
  • Is it on-topic? (yes = good)
  • How are the tags looking?
  • Is this an X/Y problem? (e.g. querent is asking some ridiculous academic question like "how can I implement RFC 1149 without a bird?", with enough specific details to make me think they're serious)
  • Are the results of failed attempts given, or are attempts just described as "failed"? (results given = good)

Except, y'know, meta answers tend to be better than this.

¹: See Does a well explained question nullify a "no research effort" downvote? and How does "proof of effort" make a question better?.

Improve this question
0

3 Answers 3

Reset to default
5

Thank you for your feedback and for taking the time to participate in this research study. This type of research was the first of its kind to help us work towards a definition of question quality on the platform. See the previous Meta post which includes context around this research study and what we are looking at. We have passed along your feedback to the team responsible, as this will help improve future research studies about question quality.

For additional context, another team also conducted a separate study recently to learn about the experiences of reviewers in review queues in order to 1) understand their process for improving questions, 2) understand the frustrating aspects of their process, and 3) understand where the most time is being spent.

This is just the beginning of our research, and we appreciate you highlighting Meta as a resource in these efforts. We will consider various research methods moving forward, including getting more community feedback.

Improve this answer
7
  • 1
    Thanks for trying to learn about these things. However, you're approaching it like Victorian anthropologists. We already have a comprehensive definition of question quality, and you're not starting from it. If your research programme continues how I think it's currently going, the result will be a new definition that you proudly share with us, that we wholly reject. Then you'll be sad, we'll be frustrated (a type of sad), and – importantly – the purpose of your research will not be achieved.
    – wizzwizz4
    Commented Jul 19 at 0:35
  • 1
    The research you're running at the moment – from what I can see of it (and I understand that I have a limited perspective) – is the sort of thing one does quite late into the research programme, when it becomes important to get fine-grained quantitative data distinguishing stated and revealed preferences. You're not at the “comprehensive map of the field” stage, nor the “explore the extents of the field” stage: you're at the “figure out where the field is” stage. (You would have a better starting point, had the company not sacked so many domain experts – but it is what it is.)
    – wizzwizz4
    Commented Jul 19 at 0:42
  • I was going to say under this SG announcement I think those 3 categories are nonsense and there's no precedent on the metas for grouping the many criteria that make good questions. In fact, if you read carefully the 3 categories have overlaps with the MRE element showing up in all 3, so at the least you should have rethought that before sending out a survey. However, since the MSO community decided not to address that obvious fact, I refrained from pointing it out myself.
    – bad_coder
    Commented Jul 19 at 1:20
  • And please notice @Sasha, after a user casts +10k close votes it's dismaying to seeing those criteria lumped together into seemingly overlapping categories. I got the sense next to no thought had been put into the categories, although it would be interesting the community just didn't put their collective mind into it, because the SG announcement lumped raw data with the orthogonal category question - that by itself defies our usual thought process because questions are supposed to "have focus" and the linked meta post lacked focus - if it weren't a staff post it would've been closed.
    – bad_coder
    Commented Jul 19 at 1:26
  • 1
    So if you want to extract the Hypernymy relations for question quality from the close vote criteria, you should separate that necessarily into its own post. Bella_Blue (your colleague from the CM team) lead the summarizing of the Question Close Reasons - Definitions and Guidance and rigorously defining/distilling those from the existing 15 yo meta corpus was quite a handful. In this case, you're attempting a new hypernymy ontology that might just not be possible, or meaningful.
    – bad_coder
    Commented Jul 19 at 1:42
  • @bad_coder RE "next to no thought": obligatory xkcd. I can kinda see how they got there from Jon Skeet's Stack Overflow question checklist, and you wouldn't necessarily know that this comment references this blog post without historical context. What they've done is very wrong, but not obviously wrong.
    – wizzwizz4
    Commented Jul 19 at 7:14
  • 1
    @bad_coder RE the MSO community not addressing this: the question was too broad. As an announcement, it's fine, but I count at least six questions in addition to the announcement proper. If I'd seen the question, it probably would've taken me… four, five hours to try to answer everything?
    – wizzwizz4
    Commented Jul 19 at 7:31
16

I really don't want this to delve too deep into "what is a good question", but, that's kinda the gist of this survey. The questions we are asked to answer in this survey follow along the same fallacies that all revisions of the ask wizard have; it treats incoming questions as either being debugging questions or not being valid, despite the fact that the most useful questions are overwhelmingly how to questions.

It's very difficult to assign a set of metrics that will work for all question types, but it should be easy to have multiple question types and allow users to decide on a case by case basis what "type" the question should be evaluated as so that correct guidance can be provided.

Improve this answer
16

Eh, I just dropped all my frame challenge responses into the various free-form text fields. I like that they're doing the survey, and hope to see more thought put into this...

But the best question of the lot I got was self-answered via an edit, and there was just no adequate rubric for that.

Jeff, if you're reading... There's no substitute for reading through a few thousand questions and seeing what gets folks excited "in the field".

Improve this answer
4
  • "self-answered via an edit" I don't understand, can you please rephrase?
    – philipxy
    Commented Jul 18 at 9:23
  • 3
    @philipxy the survey was about Q&As. One of the Q&As the survey asked Shog9 about had an answer edited into the question.
    – VLAZ
    Commented Jul 18 at 14:30
  • 6
    The survey presupposed all questions would fall into a few categories wrt evaluations of purpose, information, presentation. This one did not: the asker solved their own problem and needed help understanding what to do with that. No set of choices I might have made among the available options would have accurately reflected the problem or pointed towards its resolution.
    – Shog9
    Commented Jul 18 at 14:37
  • 3
    @philipxy in short - the survey 'missed' a real world case which didn't quite neatly fit into what the survey was asking about. The world's not perfect and there's people who put rectangular blocks in square holes. The actual point here is to an extent, it would benefit to have folks be more in touch with the platform when designing for it - to better understand edge cases like, well, someone editing in an answer like it was a forum.
    – Journeyman Geek
    Commented Jul 19 at 5:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .