31

Welcome to the Unicorn Meta Zoo, a podcast by members of the Stack Exchange community team. If you want to avoid spoilers, jump straight to the audio.

Participants

hairboat Juan M Jon Ericson

We're talking about whether or not there is such a thing as a bad question and how we might detect one if it exists.

Links

Transcript

Some users have generously volunteered to transcribe episodes on the podcast wiki. I'm extremely grateful for users who helped fill in most of the previous episode's transcript:

  • grooveplex
  • ocæon
  • MilkyWay90

Meta

It was my turn to host and I think I'm not overly biased when I say that Abby and Juan do a much better job in that role. For one thing, I was pretty anxious that we'd end the recording before we ran out of things to talk about. Not entirely unrelated: I say the word "template" in a weird way.

It didn't make the recording, but Abby started talking to her cat right after we said goodbye.

What do you think?

Take a listen and respond in the answers below.

  • 9
    I re-listened to the episode and noticed I said the top bar is always black. That is not strictly true. I still like that it's more or less the same on all the sites, however. – Jon Ericson May 21 at 15:49
  • "... to transcribe episodes ...", @Jon, did you know that you can upload a (blank) video with audio to YouTube and YT will use voice recognition to analyze the audio and produce a downloadable subtitle file? – Rob May 21 at 17:36
  • 1
    @Rob: I did. (My method is pretty rough compared to yours.) Unfortunately, folks didn't care for it. I'm feeling out whether doing that is worth the effort. If I find the time to automate the process there's no reason not to do it. – Jon Ericson May 21 at 17:42
  • 1
    That's unfortunate. --- YouTube allows collaborative editing where anyone anywhere can simultaneously edit the transcript and check each other's work; eventually producing a verbatim copy. – Rob May 21 at 17:50
  • 1
    The RSS feed, https://unicorn-meta-zoo.github.io/index.rss, sometimes results in double entries in at least one RSS reader, Mozilla Thunderbird (as the only feed out of about 21 RSS feeds). For this episode, 9 minutes apart. – Peter Mortensen May 21 at 19:05
  • @PeterMortensen: Yes. The problem likely stems from this commit. I bet you one of the entries doesn't work. Some feed readers cache episodes even when they are no longer in the feed. Sorry about that. (As always, manual steps are to be avoided.) – Jon Ericson May 21 at 19:09
  • @Rob The YouTube voice recognition algorithm is very bad at its job. – EKons May 21 at 19:25
  • 2
    @EKons: I think it depends on what you expect from it. I thought it was mostly ok, but that's not good enough if you expect to be able to understand what's going on just by reading. I did appreciate not having to start from scratch when I tried doing some transcribing. – Jon Ericson May 21 at 19:30
  • @EKons I've had better luck. I can translate German to English and get mostly readable results, that means that a video of limited interest to someone whom doesn't understand suddenly becomes much more valuable. Needless to say I find English to English works quite well. --- As I said, once 80% of the work is done it's a simple matter to be cooperative and helpful (a dozen people can work together), offering a verbatim transcript. – Rob May 21 at 19:32
  • Minor bug: it seems like the description in the podcast feed also includes “respond in the answers below”. – grooveplex May 21 at 23:50
  • @grooveplex: Good point. I think I might solve this by posting to meta first and then updating the tags on the episode and publishing the feed. That way I'd have the post number to direct people to meta. But then there would be people who would see this post before the episode was live. I'll have to think about it and come up with an automation process that works. – Jon Ericson May 22 at 0:20
  • I have a question for you if you agree that an answer can make or break a question why does SE not allow a Question to be deleted once it's been answered. – Martin Barker May 23 at 16:22
  • @MartinBarker: Well, it's complicated. For one thing, we do use answers as signal for automatically deleting questions. We also block askers from deleting their answered questions. But some questions with upvoted or accepted answers should be deleted. It's important that the community has some way to defend itself from unproductive questions. – Jon Ericson May 23 at 18:52
  • 2
    @JonEricson I was just thinking about it where there have been a couple of Questions I have asked where it has been something stupid I have had an answer that is not correct so I'm unable to delete the question even though it should not be kept on SE for the reason in the podcast the answer has nothing to actually do with what the question makes it seem like the problem is. – Martin Barker May 23 at 22:34
  • 1
    Sometimes you also get a bad-fit answer to a bad-fit question that in total is popular and informative. Hell my highest rated answer was a history of Delphi and the downfall of Borland to stupidiity, in response to a question about whatever happened to Delphi. The whole things an aberation, but Its pretty great that it all exists. Sometimes theres a lot of case for letting things slide for the greater good. – Shayne Jun 5 at 8:38
32

Thanks for the interesting discussion. You discussed how (or whether) to factor answers into the evaluation of a question as good/neutral/bad and the hazards of edge cases there. Perhaps what you're seeing is a difference between good and interesting.

We've all seen questions that the community considers poor (lots of downvotes) but that nonetheless have some outstanding answers. There's even a badge for it (Reversal). Good and/or numerous answers don't necessarily make a question retroactively good, though -- they just signal that there's something interesting here.

So do keep the interesting content available (and maybe think about affordances for finding it), but maybe you don't want to include those questions in a hypothetical (or actual) corpus of "examples of good questions". On the other hand, questions that are actually good questions, based on properties of the question, could feed into feedback to askers: here are some good examples on the tags you're using for comparison. I have no idea if y'all are thinking in that direction at all (nor am I advocating it; it's far too partly-baked for that), but if you are, I think you'll be better off focusing on the questions and not the answers they managed to collect despite question quality.

  • 15
    Simillarly there are sometimes questions which are interesting but off-topic, and they tend to gather upvotes but are ultimately closed. – Tas May 21 at 22:23
  • 5
    It's true we need different criteria in different circumstances. For A/B tests where the goal is to help users ask questions that the community is interested in answering, the quality of answers is much less important than the mere presence of answers. And if we have the stronger signal of voting, we listen to that first. I think your point about downvoted and answered questions being interesting is on-point. Arguably questions with some downvotes are more interesting than those without since controversial posts tend to gather downvotes. (Glad you enjoyed the discussion!) – Jon Ericson May 21 at 23:36
  • 11
    Answers can also indicate that the question was trivial to answer, making it an attractive easy rep source for users trying to establish themselves. On SO, these are often poor questions lacking even basic research, even often being duplicates. – jpmc26 May 23 at 6:33
  • 1
    @jpmc26 good point. If answers are factored in, probably answer length needs to be. Quick one-liners probably signal the kinds of questions you're talking about. – Monica Cellio May 23 at 13:59
  • 3
    @MonicaCellio No, the answers are often fairly long. On SO, they typically require a lengthy code block. Come to think of it, I often see high reputation users answering them as well. – jpmc26 May 23 at 15:58
  • 1
    @jpmc26: It sounds like you are bothered less by the answers themselves and more by illicite reputation gain and incentivising otherwise poor questions. – Jon Ericson May 23 at 18:58
  • Let us continue this discussion about SO question quality in chat. [Comment edited by mod to indicate what the discussion is about, so I can delete most of the moved comments.] – jpmc26 May 24 at 23:17
  • You can also write a good question, that gets downvoted to oblivion as well. – Andrew Jun 5 at 16:36
2

I don't think that judging question quality should factor answers at all.

A good / upvote-worthy question should satisfy most if not all of the following:

  • Is the problem clearly stated? We shouldn't need to derive the question parameters based on answers.
  • Is the expected outcome clearly stated? We shouldn't need to read the accepted answer's output to understand what the OP wants.
  • Do the title (<-perhaps most importantly considering search engine listings), body, and tags make it easy for researchers to find the page when the page represents the problem that they are researching?
  • Is the question unique? Stack Overflow probably suffers most from site bloat due to dupe posting, which consequently drains valuable volunteer resources.
  • Is there some indication of research? (e.g. "these pages are close, but...") This is super, super easy to prove by dropping a couple of relevant links. When linking to SE network pages specifically, this binds related SE content (win), serves as an aid to researchers by offering additional questions to chase if the current page is deemed unsatisfactory for their issue (win), and informs volunteers generally about your level of understanding (win). A possible symptom of insufficient research is the inclusion of inaccurate terminology in the post. Using correct wording makes the question easier to comprehend and improves searchability.
  • Is there context? While answers should endeavour to accommodate a general scope, context ensures that answers can conform based on sample input so that researchers can differentiate answer quality more easily.
  • Is it on-topic for the community that it is posted in? Off-topic questions, even if interesting should never be upvoted.

A question is not good merely because it can be answered.

A question is not bad merely because it has not yet received an answer.

There are heaps of Stack Overflow questions that are ~10 years old that only satisfy 2 or 3 of my above criteria, but have hundreds of upvotes. This sends a bad signal to new question posters -- they see these old questions, and think that that is the expected standard.

If a posted question only satisfies a moderate number of the above bullet points, it is a "neutral question". If it can become an upvote-worthy question, inform the OP of what is missing and/or suggest an edit.

If a question struggles to satisfy most of the above criteria, then I reckon it is downvote-worthy. A downvoted question can very possibly attract lots of valuable answers, but that result is irrelevant to judging the question's quality.

Whether or not a question is "interesting" can be safely deferred to pageview count, star count, and possibly the volume and quality of answers that it attracts. I don't believe that "interesting-ness" should be part of the evaluation.

Personally, I have a bias toward deeming questions as neutral. A question needs to be really good to get my upvote and really bad to get my downvote. This results in the community feeling relatively welcoming, while calling trash "trash", and rewarding the posters that employ due diligence and bother to craft exceptionally awesome content for SE.

I also don't automatically downvote a question for being a duplicate. If the OP has obviously toiled to craft a good question, but just didn't know where to look, I'll vote to close and make no up/down vote.

  • Much of this answer sounds really familiar. Is it copied from a FAQ or another meta post somewhere? – Rand al'Thor Aug 24 at 6:20
  • All of it was from my own brain. I was actually worried that I left out a few criteria because I was writing it on the fly. – mickmackusa Aug 24 at 6:24
  • 1
    @Randal'Thor it sounds familiar because it is paraphrasing what users have been saying for the last six or seven years! – Mari-Lou A Aug 24 at 11:26
  • @Mari-LouA I know that; I meant the exact phrasing sounded familiar. Deja vu perhaps. – Rand al'Thor Aug 24 at 11:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .