The Anatomy of a Hot Question

Question

Here is the traffic since The Workplace was created:

These are the three questions that caused those huge spikes in traffic:

1. Is it rude to leave an interview early if you have already made your decision?
2. How should I deal with an employee who has slept with my wife?
3. What is a 'friendly' way to let managers know that having good developers is a privilege?

These questions alone account for some of the highest viewed questions on our site (#3, #2, and #5 respectively).

So What's the Issue?

More visits should be a good thing. More exposure should be a good thing. But they aren't right now for three reasons:

1. New Users drown out community moderation
2. As a result these questions make bad examples
3. Community moderation burdens increase

These are all quantifiable harms to growing communities. And they should be avoided.

Voting Volume Increases 233%

The week prior to one of these hot questions, on average (all three cases) we have an average of 128 votes/day. During the week of a hot questions, that more than doubles to almost 300 votes/day. That is a lot of additional votes.

Most of those are upvotes. On TWP in general, our upvote/downvote ratio is 13.6. For these hot questions, the upvote/downvote ratio is a whopping 30.4.

People come in, make tons of upvotes, and any direction the community wants to give is lost in the noise.

Question Volume Increases 190%

The question volume also increases by 190%, from 17.2 posts/day to almost 33 posts per day. The downvote ratio on these posts is much higher -- the week prior to a hot question we have an average of 0.39 downvotes per post made. The week during a hot question it raises to 0.47. The week after it rises to 0.67 downvotes/post.

The data explorer does not include deleted questions, which would further skew the numbers.

People click through the hot questions, see mediocre questions and answers with tons of upvotes, and take that as carte blanche to start posting their own

12 Reserve Active Members Required

Peak voting during one of the hot questions is over 600 votes per day. On the week before we peak around 250. That means we have to make up 350 additional votes per day. Users have a limit of 30 votes per day. That means we need 12 additional active users casting max votes per day to try to make a dent in the votes from new users flooding in.

That requires time. Especially when the volume of closed and deleted questions and answers increases due to the drop in quality from the new users.

Even if we tried, the flood is overwhelming compared to the ability of a community to self-moderate.

There are Unquantifiable Issues Too

A quick check of our meta will find times where regular users wonder why we close so many questions. When the community is stretched thin voting, protecting, closing, deleting, and commenting on posts by new users, making aggressive edits and working with users to teach them the rules and help them improve their posts is a luxury that feels like it can wait.

Seeing one question bring in so many users often spawns successive hot questions which end up compounding the problem because even poor answers increase hotness score. The constant flood of new users and watching the usually clean review queue shoot up as well as the number of flags takes away moderator resources as well as contributes to community moderation burnout.

A Small Request

This last storm of traffic was mitigated (slightly) by me popping in the Tavern of the Meta and getting a dev to protect the question before it got out of hand. And that worked great. Only...

1. Is that really the best use of the dev's time?
2. Should we really expect your average user to pop over to meta.so chat to try to solve a problem on a beta site?

Community moderators should be aware of this issue, and what it does to community moderation. If a hot question pops up on a smaller site, I plead with whoever is awake and around to just plant yourself in that site's chat and be available to take action if the community moderators need it. Beta sites have only a few mods who aren't generally awake all the time (though I don't think jmort sleeps). Just the gesture to say, "Hey, we are here to help" would be a morale booster to people who are overwhelmed by janitorial duty.

Of course, this could be helped by nipping the problem in the bud, but in the meantime, community mods, please realize that us unpaid community volunteers could use your help sometimes, and would really appreciate it.

Ben's Query

Ben says in the comments:

What would be interesting to know is how many users, on average, stuck around and continued to be a contributing member of the community after arriving in this fashion. It could be that the number of new users you gain is more than worth the extra bit of moderation for a few days.

I took a look at the new users whose first post was an answer on one of those first two hot questions. With the help of many users, I was able to create a query that looks at new users who joined via those questions. Unfortunately, data.se doesn't look at deleted posts. There are 14 additional users with deleted questions, 3 of which are unregistered accounts. So altogether we have 24 new users who joined through these posts directly.

Here is what those users look like:

　　　　　　　　　　　　　　　　　　　　　　^Reputation

^{The numbers at the bottom of the chart on the X-axis represent the number of users who have reached that reputation level. 500 is the cutoff for closing on beta sites}

only 33% of all users who joined through these questions are active. There are only 2 active 500+ members who can help with community moderation (only one of them has helped). 42 reviews is not comparable to the increase in moderation required for these questions.

These questions do not seem to bring long-term users with good contribution to the community.

Answer 1

As a mod on UX.stackexchange (a graduated site) we get the same issue. Something hits the HQ top 3 and we get loads of comments, 'answers' and all sorts and invariably the solution is to delete a bunch of posts and protect the question.

So my suggestion is — when a question hits the top 3 of the HQ list then automatically protect it so that <10 rep users can't post directly on that question.

'But we want to encourage people to post, that's the whole point of the HQ list!' I hear you cry. I disagree. We want to encourage people to the site not just to that question. By protecting a question it encourages users to go and post a useful question/answer elsewhere on the site. If they do that, their rep will increase (if they've posted something useful) then they can contribute to the hot question itself. Hopefully by posting a useful question / answer they will then have done a bit more studying into what the site is all about, we'll get extra useful posts and some new users.

Heck, these questions usually get protected anyway, why not just make it automatic?

(I'm aware that there's probably more to it than 'just set it protected automatically'; nothing is ever that simple, but I think it bears consideration at least)

Answer 2

This isn't really an issue of moderation, this keeps happening even on sites where there's at least two mods active at any given time. Questions can stay on that list longer than most people can stay awake - there will always be times where these hit during a gap. I don't think it can be avoided.

I'm not going to go so far as to call this a problem. However, it's something that we should probably address. There's quite a few users on every single site that we have who don't appreciate the party bus driving slowly through their neighborhood with the music up full blast, and they definitely don't appreciate picking up all of the bottles and beer cans after it leaves.

Get on the hot questions list, tweeted and then mentioned on Reddit and things get quite interesting for non-technical sites. On sites where the average person can comprehend every post for the most part - participation does tend to explode.

My proposed solution, which is still very much on the drawing board, allows for moderators or community managers to evict a question from the list provided that:

• There is definitely evidence of mayhem (as seen by vote/visit, comment/visit, question being protected while featured, and other things)
• The question has been featured for at least [n] hours (this wasn't just Jeff making fun of a PHP question on Twitter again to get it on the list, and it will soon die down in a sea of 'meh')
• Other criteria I haven't yet thought of (this is very much just an idea at this point, and the bullets above are also not carved in stone)

The problem is, this is a bit of a technical nightmare. That list is heavily cached because the query that goes into it is very expensive (or at least it was the last time that I heard Jarrod talking about it). Moderators would have to literally be able to evict it right from cache, which presents an interesting problem of a hot questions list that shrinks soon after being compiled.

You're right in expecting help to show up when it's needed - which is part of the reason why the community managers have moderator (and beyond) access on every site. We're also moderators on any site whenever the site needs extra hands. During the week, heck, even half of the weekend we have pretty much 24 hour coverage thanks to time zones.

Doing what you did is precisely what you should have done, and pretty much all you could have done. If your site is having problems and none of your mods happen to be around / awake at the time - any Stack Exchange employee can help you either by jumping in, or locating someone that can. Do not hesitate to reach out to anyone that works here if your site is in trouble and you need help.

Ideal, no - but it can work until something more tenable is put in place.