77

I've been quite vocal about the fact that we're overhauling post blocks, and dropped some hints as to how we're contemplating improving them. The last few weeks of my professional existence has gone into analyzing what we don't like about them, re-visiting what we hoped to accomplish using them, and coming up with a system that better supports those goals.

What we have was good, at the scale that it was introduced.

Our post blocks need to catch up with the times. Let's reiterate briefly, here's what we don't like:

  • They don't slow you down fast enough at the point where you really need them to. You need to fall on your face a few times.
  • Once tripped, due to the scale of voting that we see today, they can be practically impossible to escape. Some questions simply can't be improved and subsequently up-voted because you fell on your face in wildly off-topic ways a few times.
  • It's easier to delete your account than follow the advice that we give you.

The third item in particular led to a big problem with recidivism, or folks that just entered this maddeningly myopic and dysfunctional cycle of throwing themselves at a wall until we blocked them, then rinsed and repeated again. We put a stop to it mostly, but that's as much of a stop-gap as it is a test to see how effective increased rate limiting would be. The jury is still out on how effective it has been, we're optimistic, needs more time.

Now, onto what we really wanted to accomplish - that's not difficult to state. We want to ensure that Stack Overflow maintains the level of quality that made it the phenomenal success that it is. Tough love, as it were, seemed the easiest and most logical way to make those that can ask better questions do so, and those that simply can't to go elsewhere.

We over-simplified the problem.

Yet, something about the way we block has always bothered me, and I couldn't quite put my finger on it until recently. One of our very public secret agendas has been to trick programmers into becoming better communicators through better writing. By blocking as we do, we're stopping folks that have this potential from pursuing it and prospering. Stack Overflow tends to make people better at more things than programming, if you tough it out.

We need a system that repels help vampires, while helping inexperienced programmers - the folks that can be helped - ask better questions without discouraging them to the point that they see account deletion as the path of least resistance.

Let's bolt some airbags on the front of the question block mechanisms to slow people's acceleration and the force with which they hit them. To do this, we need to look at how well you ask questions overall, and how well you've asked questions recently. The first question doesn't count, because we're programmers and enjoy testing gravity with our faces and some pavement from time to time.

This is where it gets ... tricky, and I'd love some input.

In a single sentence, a question block is the system preventing you from asking questions based on your history of asking questions, until you manage to improve your questions sufficiently to earn some up-votes, or provide quite a few up-voted answers. Basically "You seem to suck, show us that you don't and we'll let you ask again".

Imagine seeing this after asking two questions that didn't go over so well:

The experience you had with your last two questions wasn't what we hoped it would be. Why not take some time and browse other questions tagged (tags) that have been well received? You can then come back tomorrow and try again.

Grr, that stupid site! you grumble as you search around some more, and come back the next day to try again. This time, you do slightly better and manage to not get down-voted, and possibly up-voted. If you do that, we stay out of your way as consistently as you ask questions that don't make people scream.

If you don't - then you get one question every few days, one question per week, three per month and then ultimately:

Sorry, we're no longer accepting questions from this account

To be clear, this doesn't allow more low quality questions in, keep in mind, folks are heavily rate-limited before that particular side of it actually kicks in.

Basically, the system trains on how we'll you've done overall (either your entire account history, or the last 45 days for newer accounts, dropping the most negatively scored), and how well you've done recently (the last 15 days). That gets us the following:

  • Number of questions asked in the last 45 days
  • Average score of your questions in that time
  • Number of questions asked in the last 15 days
  • Average score of your questions in that time
  • Average time elapsed between questions in the last 45 and 15 days

People that ask good questions also tend to ask fewer questions, so what we're looking at is pretty simple:

  • Does user ask good questions? Nothing more to do
  • Does user have a history of bad questions?

    • Do they seem to be improving based on what we can see? Let them ask more questions, with limits commensurate with recent quality contributed
    • Are they not improving? Limit them, in a manner commensurate with the way they're being received.

Sounds easy, huh? Not quite.

Now, I've got numbers and such in mind, and much like the existing quality blocks, we won't be revealing the exact mechanics behind how rate limiting works or it just turns into a case of carrots on sticks.

However, a chief goal is to not put off someone that could actually be a good contributor to the point that they just get frustrated and quit, while souring the metaphorical milk enough to make the help vamps go somewhere else.

Those that stay, and prosper, have done so because they've treated questions as a resource that is not infinite, and made them count when asking.

How would you slide these scales in a manner not likely to put off potentially good contributors, but annoyingly enough to drive the vamps to the hills? What pitfalls do you see with such a system? What else should we be looking at?

The last thing to keep in mind is other work going on to raise the perceived quality of most new questions, so this is one of many moving pieces. Unlike the current block, this doesn't essentially 'make or break' the quality of incoming questions. Oh, and - this applies only to sites that have 'big city' problems that come with larger scale.

Stack Exchange Quality Improvement Project

11
  • 4
    what are question blocks in a single sentance please?
    – juergen d
    Commented Jul 15, 2014 at 13:47
  • 11
    A question block is the system preventing you from asking questions based on your history of asking questions, until you manage to improve your questions sufficiently to earn some up-votes, or provide quite a few up-voted answers. Basically "You seem to suck, show us that you don't and we'll let you ask again".
    – user50049
    Commented Jul 15, 2014 at 13:49
  • 5
    The rate limiting strikes me as a solution that sounds great and friendly and positive from our perspective. But if you're at the receiving end of it, you have your "urgent" question to ask, and it tells you to come back in two days ... would they not simply still create a new account?
    – Bart
    Commented Jul 15, 2014 at 13:52
  • 3
    @Bart they will... And they'll still be told to wait two days. Commented Jul 15, 2014 at 13:54
  • 4
    @Bart Surprisingly, not always. If you're keen enough to keep cycling through tor proxies and email addresses and such, you're generally well-off enough to ask a decent question. Most just request deletion and come back with credentials they've used before (though they do vary them sometimes, we eventually see 'em all). Limiting is also in place for brand new users with no history now, that's been around for quite a while.
    – user50049
    Commented Jul 15, 2014 at 14:18
  • 1
    Also, @Bart - we're going to be looking at how many low-quality questions recently came from your network as we route new questions differently in a review triage - so making new accounts isn't really going to help if you get no visibility on your questions until someone else either blesses or edits them. It's going to be increasingly difficult to work around. It's frankly easier to just put more thought into your posts than try and get around the things forcing you to do so.
    – user50049
    Commented Jul 15, 2014 at 14:27
  • Sounds very good and very much needed, but I would also take into account answers and other helpful actions (flagging, reviewing) in the calculation. For example if user answered 10 questions, with total of 20 upvotes and 5 downvotes, then he ask 3 questions, all of them get -1 to -5 it wouldn't be fair to ban or rate limit just yet. Commented Jul 15, 2014 at 14:27
  • 1
    @ShadowWizard Get that in an answer if you don't mind. I am considering it to some extent but I'd love to see you run with that a bit more.
    – user50049
    Commented Jul 15, 2014 at 14:29
  • Done, @Tim. :-) Commented Jul 15, 2014 at 14:31
  • related: Why won't the system allow me to ask questions for several days?
    – gnat
    Commented Sep 20, 2014 at 20:44
  • system seems to penalize migrated answered questions the same as deleted ones, is this intended?
    – gnat
    Commented Oct 30, 2014 at 21:52

6 Answers 6

29
+50

Yes, I think a gradual rate-limiting, with safeguards for recidivism, will be a much better solution in the long run than our current question bans. After observing the absolute worst trash coming in to Stack Overflow over the last several months, it seems clear to me that a large portion of our worst questions come from people who repeatedly work around the question ban. A single terrible user hits the question ban after around 3 questions. A user who works around this can spew a near unlimited number of these.

The two classes of abusive users who need to be addressed are:

  • Those who evade a question ban by creating new accounts
  • Those who evade a question ban via voting fraud (sock puppets, voting rings)

It's an open secret that it's trivial to evade a question ban by simply creating a new account. Most people I've seen don't bother to delete their old account when doing this, they just use other credentials. If we're relying on restricting people who delete their accounts and reuse their credentials to create their new ones, I suspect we'll miss almost all of these abusive askers. We need to be able to somehow rate-limit across all of these new accounts. People almost always create their new accounts immediately after they are blocked from asking on their previous one, and do so at the same location, so we have some data points for tracking these new accounts.

The second group of people are more troubling, because they can circumvent a question ban completely and ask questions forever. Over the last two months, almost every single person I caught running sock puppets were doing so to vote themselves out of a question ban. In many cases, it was the original, question-banned account that was being used for this. Voting rings are also commonly employed to evade the question bans.

Now moderators can step in and remove these sock puppets, but that requires us to see them and identify what is happening. By that time, these askers can spew a lot of absolute garbage on the site and have it be artificially upvoted. It would be a tremendous help if the system could somehow identify this activity. Again, this always follows the same pattern: someone gets question banned, then either magically is lifted from that ban by a series of votes from accounts at their location, or a new account at that same location is created after one is banned and the latter is voted for by accounts at that location.

I am convinced that a system to inhibit both classes of users who work around question bans will have an immediate and noticeable effect on the question quality on Stack Overflow (if not elsewhere).

Unrelated to the above, when weighting items for the question rate limiting, perhaps we should take into account question closure reasons. On Stack Overflow, true help vampires are more likely to have their questions closed as "too broad" or "unclear what you're asking". Maybe by combining the close reason with voting, we could have a better heuristic for identifying those who are more likely to be persistent problems. Folks asking duplicate, off topic, or opinionated questions tend to not be long-term problems, in my observation.

7
  • 3
    We're not relying on the credentials being the same, we're going to be 'highly skeptical' of networks that produce a ton of crap. Note, this isn't something we'd outright block on from a new user, but they'd need a few folks to sign off on their question before it got any visibility outside of /review. And yeah, I'm also looking at the reasons stuff was closed both to produce better help for folks that hit this, and influence the set of dip switches we're building that determine where your question goes before folks see it, if the system thinks it might be crap.
    – user50049
    Commented Jul 15, 2014 at 21:46
  • 1
    As the system temporarily records who has up voted a question (so you can retract the vote), could up voters for poor quality questions be added to a list of suspected sock puppets? Once enough evidence was accumulated the sock puppets could be dealt with semi automatically. Or is something like this already don?
    – Raedwald
    Commented Jul 15, 2014 at 21:47
  • For the folks that love cycling through a dozen exit nodes, well, we already have a means of dealing with that quite well :) Thanks for your input Brad, I'm re-thinking a few things now.
    – user50049
    Commented Jul 15, 2014 at 21:47
  • @Raedwald separate feature, but yeah, warning you based on a short sampling of your recent votes is on the board. Feeding that into the bits we flip on quality scores might be cheap enough to do. This isn't about sussing out socks for this sort of thing, this is more about making them ... ineffective.
    – user50049
    Commented Jul 15, 2014 at 21:49
  • @TimPost - Yeah, the Tor or proxy folks generally aren't the ones doing this. You have to apply some level of effort to do that. These are just "I need my questions answered to do my job" types, who only find the path of least resistance to keep asking questions. When they're naming their sock puppets the same thing as their main account, you know they're not trying too hard to hide. Like Shog9, I've "abused" the anti-trolling system against a few of these, and it worked wonders in slowing them down. Commented Jul 15, 2014 at 21:54
  • 6
    @BradLarson Don't discount training manuals from sweatshops that make cycling through exit nodes a cinch with the help of your company IT guru just so Acme, Inc has employees that can ask questions. Because we're pretty sure that's what's happening in some cases, so I've got to consider that.
    – user50049
    Commented Jul 15, 2014 at 21:56
  • @TimPost, Does this problem with sweatshops collate with the country the user is in? If so that should be anther input. Commented Sep 24, 2014 at 22:00
20

At the risk of over-simplifying the problem again, only in a different way, it seems like there are two kinds of users getting blocked:

  • help vampires
  • Others (people who can improve with a little encouragement)

Would it help if you could just positively identify help vampires, then apply a different set of (unpublished) criteria for staki...er... blocking them? A few tell-tale signs:

  • Do they delete questions then quickly post another one?
  • Do they rarely, if ever, vote?
  • Do they rarely, if ever, edit their questions after a comment has been posted?

Looking at this from the Other perspective, a different pattern I've noticed is users getting question blocked after one question is pile-on downvoted. It almost always turns out that they had several slightly below average (-1 score) questions that contributed to the ban (as opposed to one or two other highly downvoted posts). These are almost always users that can be encouraged to improve.

Rather than blocking users when they reach the average question score threshold, maybe the system could check to see if their average score is still too low after one or two outliers are removed? Users who are suffering from one or two bad questions could be given a warning and some tips on improving their existing questions, rather than being blocked outright from asking more questions.

4
  • 4
    In my experiences help vampires are the most avid voters, not users that don't vote at all. They very quickly learn that upvoting anyone and everything in their path and accepting answers right away makes would be answerers super happy and makes them want to answer their questions all the more often, because these answerers learn that they'll get tons of free rep "just for trying" regardless of answer quality. People taking the time to actually test out answers, vet them, understand them, etc. tend to, in my experiences, vote less quickly and less often.
    – Servy
    Commented Jul 15, 2014 at 21:24
  • 3
    "Do they... edit their questions after a comment" Checking whether they edit questions after they have been closed might be more significant (ignoring closed-as-duplicates).
    – Raedwald
    Commented Jul 15, 2014 at 21:37
  • Thank you Bill! I'm adjusting a thing or two based on a thing or two that you brought up.
    – user50049
    Commented Jul 15, 2014 at 21:38
  • 2
    @Servy It would be interesting to look at some data on this. I often see complaints of people asking questions and not upvoting or accepting answers. It could be that we're both right, and a characteristic of the help vampire is that they either never vote or always vote up every answer. It might be worth looking into both cases on a set of users who are blocked. Commented Jul 15, 2014 at 22:10
18

It seems that duplicates count the same as other closure reasons for this system. The asker of the question I linked there has gotten the "Some of your recent questions were not well-received" warning based on two duplicate closures. But we like well-asked questions that get closed as duplicates; they can be good for the site, signposts, yadda, yadda. Old discussion.

It therefore seems like a bad idea to castigate askers for posting a good question that turns out to already be answered. I'd suggest that dupes be special-cased to count less for the rate limits/block, or -- possibly better -- only count if also downvoted.

Downvotes on dupes, in my experience, are the "I can't believe this is being posted again" signal (deletion signals that, too). They indicate that the asker is wasting other people's time and probably should be rate limited. On the other hand, an upvoted dupe often indicates a novel or at least perfectly clear and detailed expression of a problem that someone else just happens to have posted before.

Duplicates can be questions we want, despite being closed, and should be treated as such by this system.

2
  • I agree completely. I have seen questions closed as dupes because somewhere in the linked question, it answers the dupe; like "what is xyz?" being closed as a dupe of "why is xyz used for ___?". These are still perfectly valid questions and the user shouldn't be discriminated for it.
    – kettlecrab
    Commented Oct 21, 2014 at 2:13
  • A more common case is that the question is closed as a duplicate of a question which answers it somewhere, somehow in the text. It might answer it indirectly, or not as the main focus of the question, or it may be given in one person's answer and not the other. It makes sense that questions can be closed as dupes in this way, but certainly, it isn't like the duplicate question had no research involved.
    – kettlecrab
    Commented Oct 21, 2014 at 18:12
16

Sounds very good and very much needed, but something appears to be missing.

I would also take into account answers and other helpful actions (flagging, reviewing etc) in the calculation.

For example if user answered 10 questions, with total of 20 upvotes and 5 downvotes, then he asks 3 questions, all of them get -1 to -5 it wouldn't be fair to ban or rate limit just yet, as will most likely happen.

Numbers are of course highly flexible, but that's the general idea.

15
  • 2
    This is reaching back to the sort of 'general / global citizenship score' Jeff was talking about some time ago. It's not a bad idea to calculate something like that for this very specific purpose, just not showing it.
    – user50049
    Commented Jul 15, 2014 at 14:48
  • Sure, showing it would likely cause same trouble the accept rate did. But I think the current question ban algorithm is using answer score, isn't it? So it's not a new idea. Commented Jul 15, 2014 at 14:50
  • 13
    Agree with you on the whole except for reviewing being taken into account. If there is anyone I want to discourage from reviewing, it's the folk who can't even keep themselves out of the question ban.
    – OGHaza
    Commented Jul 15, 2014 at 16:01
  • 2
    @OGHaza If they're terrible at reviewing, they'll get banned from that too. Commented Jul 15, 2014 at 17:47
  • @OGHaza someone might be not so good in programming or have bad question asking skills, yet invest time in improving grammar of questions, fixing code formatting all around etc. Not much, but still it means he cares. Commented Jul 15, 2014 at 18:46
  • 1
    To put OGHaza's comment another way, someone who isn't cognizant of the poor fit of his own content is unlikely to be a good judge of others' content.
    – jscs
    Commented Jul 15, 2014 at 18:48
  • 2
    @ShadowWizard: That's editing, not reviewing.
    – jscs
    Commented Jul 15, 2014 at 18:48
  • 1
    @JoshCaswell true, blonde moment. Anyway, taking answers into account is good enough, and possibly helpful flags as well. The rest can be safely ignored, if the team decides it's not worth it. Commented Jul 15, 2014 at 18:52
  • 1
    I should clarify, too: I agree that editing should be counted as a positive contribution. (Well, as long as it's thorough and correct &c., but that's a whole 'nother barrel.)
    – jscs
    Commented Jul 15, 2014 at 18:54
  • 3
    @JoshCaswell with robo approvers out in the open, it's hard to tell when someone really suggests good edits so might be better not taking the risk. :/ Commented Jul 15, 2014 at 18:54
  • True, @ShadowWizard; it's a big barrel to tackle.
    – jscs
    Commented Jul 15, 2014 at 18:57
  • I think that if you compare the question scores with the user's answer scores, the answer scores should be weighted downwards because, afterall, it's about asking questions, not answering them.
    – kettlecrab
    Commented Oct 21, 2014 at 2:02
  • 1
    @FizzledOut true, good point. Thanks! :) Commented Oct 21, 2014 at 6:43
  • I edited hundreds of posts with the restriction, trying to get out of the ban. I sighed and made another account. When they were merged, the ban was lifted. Evidence that people can changed. Commented Oct 18, 2016 at 0:07
  • @uoɥʇʎPʎzɐɹC editing is not a sign you can ask or answer properly. For all we know, you can just edit 1000 "i" to "I" etc. Commented Oct 18, 2016 at 6:14
2

If you believe that respect for testing gravity and bolting airbags in front of the block is important, then the criteria that "first question doesn't count" would better be rephrased to something like "first bad question doesn't count".

To stay on a safer side, system would better assume that all first well received questions prior to first bad one are just luck.

  • Imagine a junior developer working in a mature project having detailed instructions on how to research the issues prior to falling back to Stack Overflow and how to present their question and research they have done at SO if needed. Such a guy can easily ask 5... 10... 20... good questions not because they understand how site works mind you but because they simply follow instructions in their project. Imagine one day they decide to ask something of their own, something not related to their project, not covered by instructions, like boat programming - that would be the day when they really start learning about the site.

Don't get me wrong, I think there is a far better chance of the opposite, that is one asks good questions from day one because they understood how things work. My point is, for the automatic system it would be just safer not to assume this and act as if these first good questions were accidental.

The way you describe how block is intended to work, it looks like unless there is a solid evidence that user learned how things work when they ask bad question, system would better assume that they don't know. From that perspective, first good questions don't count as evidence of learning.

2

I think this is a great idea over all, but I see no reason to ever resort to a permanent question ban. What's the harm in continuing to dangle the carrot? Yeah, maybe the user will need to wait a year or two or to get another try, but why force them to give up?

1
  • 2
    In implementation it's never quite permanent. It's .. long, depending on how long it takes for your lowest-scored posts to fall out of the scope of a query, but it's never completely permanent. It could take a year to lift, though. But, by the time you hit that, you've been warned and limited at least five times now.
    – user50049
    Commented Oct 21, 2014 at 10:20

You must log in to answer this question.