Unicorn Meta Zoo #3: How do we grade questions?

Question

Welcome to the Unicorn Meta Zoo, a podcast by members of the Stack Exchange community team. If you want to avoid spoilers, jump straight to the audio.

Participants

We're talking about whether or not there is such a thing as a bad question and how we might detect one if it exists.

Links

• How is question quality measured in A/B tests?
• Ask a question template v1 experiment results
• “Ask a question” wizard prototype

Transcript

Some users have generously volunteered to transcribe episodes on the podcast wiki. I'm extremely grateful for users who helped fill in most of the previous episode's transcript:

• grooveplex
• ocæon
• MilkyWay90

Meta

It was my turn to host and I think I'm not overly biased when I say that Abby and Juan do a much better job in that role. For one thing, I was pretty anxious that we'd end the recording before we ran out of things to talk about. Not entirely unrelated: I say the word "template" in a weird way.

It didn't make the recording, but Abby started talking to her cat right after we said goodbye.

What do you think?

Take a listen and respond in the answers below.

Answer 1

Thanks for the interesting discussion. You discussed how (or whether) to factor answers into the evaluation of a question as good/neutral/bad and the hazards of edge cases there. Perhaps what you're seeing is a difference between good and interesting.

We've all seen questions that the community considers poor (lots of downvotes) but that nonetheless have some outstanding answers. There's even a badge for it (Reversal). Good and/or numerous answers don't necessarily make a question retroactively good, though -- they just signal that there's something interesting here.

So do keep the interesting content available (and maybe think about affordances for finding it), but maybe you don't want to include those questions in a hypothetical (or actual) corpus of "examples of good questions". On the other hand, questions that are actually good questions, based on properties of the question, could feed into feedback to askers: here are some good examples on the tags you're using for comparison. I have no idea if y'all are thinking in that direction at all (nor am I advocating it; it's far too partly-baked for that), but if you are, I think you'll be better off focusing on the questions and not the answers they managed to collect despite question quality.

Answer 2

I don't think that judging question quality should factor answers at all.

A good / upvote-worthy question should satisfy most if not all of the following:

• Is the problem clearly stated? We shouldn't need to derive the question parameters based on answers.
• Is the expected outcome clearly stated? We shouldn't need to read the accepted answer's output to understand what the OP wants.
• Do the title (<-perhaps most importantly considering search engine listings), body, and tags make it easy for researchers to find the page when the page represents the problem that they are researching?
• Is the question unique? Stack Overflow probably suffers most from site bloat due to dupe posting, which consequently drains valuable volunteer resources.
• Is there some indication of research? (e.g. "these pages are close, but...") This is super, super easy to prove by dropping a couple of relevant links. When linking to SE network pages specifically, this binds related SE content (win), serves as an aid to researchers by offering additional questions to chase if the current page is deemed unsatisfactory for their issue (win), and informs volunteers generally about your level of understanding (win). A possible symptom of insufficient research is the inclusion of inaccurate terminology in the post. Using correct wording makes the question easier to comprehend and improves searchability.
• Is there context? While answers should endeavour to accommodate a general scope, context ensures that answers can conform based on sample input so that researchers can differentiate answer quality more easily.
• Is it on-topic for the community that it is posted in? Off-topic questions, even if interesting should never be upvoted.

A question is not good merely because it can be answered.

A question is not bad merely because it has not yet received an answer.

There are heaps of Stack Overflow questions that are ~10 years old that only satisfy 2 or 3 of my above criteria, but have hundreds of upvotes. This sends a bad signal to new question posters -- they see these old questions, and think that that is the expected standard.

If a posted question only satisfies a moderate number of the above bullet points, it is a "neutral question". If it can become an upvote-worthy question, inform the OP of what is missing and/or suggest an edit.

If a question struggles to satisfy most of the above criteria, then I reckon it is downvote-worthy. A downvoted question can very possibly attract lots of valuable answers, but that result is irrelevant to judging the question's quality.

Whether or not a question is "interesting" can be safely deferred to pageview count, star count, and possibly the volume and quality of answers that it attracts. I don't believe that "interesting-ness" should be part of the evaluation.

Personally, I have a bias toward deeming questions as neutral. A question needs to be really good to get my upvote and really bad to get my downvote. This results in the community feeling relatively welcoming, while calling trash "trash", and rewarding the posters that employ due diligence and bother to craft exceptionally awesome content for SE.

I also don't automatically downvote a question for being a duplicate. If the OP has obviously toiled to craft a good question, but just didn't know where to look, I'll vote to close and make no up/down vote.