What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

Here is the traffic since The Workplace was created:

Quantcast Traffic Report for TWP

These are the three questions that caused those huge spikes in traffic:

  1. Is it rude to leave an interview early if you have already made your decision?
  2. How should I deal with an employee who has slept with my wife?
  3. What is a 'friendly' way to let managers know that having good developers is a privilege?

These questions alone account for some of the highest viewed questions on our site (#3, #2, and #5 respectively).

So What's the Issue?

More visits should be a good thing. More exposure should be a good thing. But they aren't right now for three reasons:

  1. New Users drown out community moderation
  2. As a result these questions make bad examples
  3. Community moderation burdens increase

These are all quantifiable harms to growing communities. And they should be avoided.

Voting Volume Increases 233%

The week prior to one of these hot questions, on average (all three cases) we have an average of 128 votes/day. During the week of a hot questions, that more than doubles to almost 300 votes/day. That is a lot of additional votes.

Most of those are upvotes. On TWP in general, our upvote/downvote ratio is 13.6. For these hot questions, the upvote/downvote ratio is a whopping 30.4.

People come in, make tons of upvotes, and any direction the community wants to give is lost in the noise.

Question Volume Increases 190%

The question volume also increases by 190%, from 17.2 posts/day to almost 33 posts per day. The downvote ratio on these posts is much higher -- the week prior to a hot question we have an average of 0.39 downvotes per post made. The week during a hot question it raises to 0.47. The week after it rises to 0.67 downvotes/post.

The data explorer does not include deleted questions, which would further skew the numbers.

People click through the hot questions, see mediocre questions and answers with tons of upvotes, and take that as carte blanche to start posting their own

12 Reserve Active Members Required

Peak voting during one of the hot questions is over 600 votes per day. On the week before we peak around 250. That means we have to make up 350 additional votes per day. Users have a limit of 30 votes per day. That means we need 12 additional active users casting max votes per day to try to make a dent in the votes from new users flooding in.

That requires time. Especially when the volume of closed and deleted questions and answers increases due to the drop in quality from the new users.

Even if we tried, the flood is overwhelming compared to the ability of a community to self-moderate.

There are Unquantifiable Issues Too

A quick check of our meta will find times where regular users wonder why we close so many questions. When the community is stretched thin voting, protecting, closing, deleting, and commenting on posts by new users, making aggressive edits and working with users to teach them the rules and help them improve their posts is a luxury that feels like it can wait.

Seeing one question bring in so many users often spawns successive hot questions which end up compounding the problem because even poor answers increase hotness score. The constant flood of new users and watching the usually clean review queue shoot up as well as the number of flags takes away moderator resources as well as contributes to community moderation burnout.

A Small Request

This last storm of traffic was mitigated (slightly) by me popping in the Tavern of the Meta and getting a dev to protect the question before it got out of hand. And that worked great. Only...

  1. Is that really the best use of the dev's time?
  2. Should we really expect your average user to pop over to meta.so chat to try to solve a problem on a beta site?

Community moderators should be aware of this issue, and what it does to community moderation. If a hot question pops up on a smaller site, I plead with whoever is awake and around to just plant yourself in that site's chat and be available to take action if the community moderators need it. Beta sites have only a few mods who aren't generally awake all the time (though I don't think jmort sleeps). Just the gesture to say, "Hey, we are here to help" would be a morale booster to people who are overwhelmed by janitorial duty.

Of course, this could be helped by nipping the problem in the bud, but in the meantime, community mods, please realize that us unpaid community volunteers could use your help sometimes, and would really appreciate it.

Ben's Query

Ben says in the comments:

What would be interesting to know is how many users, on average, stuck around and continued to be a contributing member of the community after arriving in this fashion. It could be that the number of new users you gain is more than worth the extra bit of moderation for a few days.

I took a look at the new users whose first post was an answer on one of those first two hot questions. With the help of many users, I was able to create a query that looks at new users who joined via those questions. Unfortunately, data.se doesn't look at deleted posts. There are 14 additional users with deleted questions, 3 of which are unregistered accounts. So altogether we have 24 new users who joined through these posts directly.

Here is what those users look like:

Graph of New Users by Rep and Activity

                      Reputation

The numbers at the bottom of the chart on the X-axis represent the number of users who have reached that reputation level. 500 is the cutoff for closing on beta sites

only 33% of all users who joined through these questions are active. There are only 2 active 500+ members who can help with community moderation (only one of them has helped). 42 reviews is not comparable to the increase in moderation required for these questions.

These questions do not seem to bring long-term users with good contribution to the community.

share|improve this question
    
Am I right to assume this problem can affect only sites in beta phase? –  Shadow Wizard Jan 27 at 9:37
3  
@ShadowWizard directly, this problem can affect any site smaller than trilogy - eg one can see similar "flash in the pan" observations in this coverage of Programmers history. Indirectly, it affects larger sites, too, as highly visible proof that even low quality answers have good chance to gain reputation spreads across whole SE network –  gnat Jan 27 at 9:57
    
@gnat the problem described by Yannis is much more complex than hot questions, but you got a point. Still, my suggestion is to omit beta sites from appearing in the hot questions list, at least in the side bar. –  Shadow Wizard Jan 27 at 10:05
    
@ShadowWizard as far as I can tell what you suggest means more exposure to hot list damage to smaller graduated sites like Programmers (and Math and Code Golf...), and, soon to Workplace (yes, again) - because it is likely to graduate in 1-2 months. What do you think will happen at TWP then, when they get exposed in that list again, only with moderating privileges of active regulars lowered from beta to graduated levels? –  gnat Jan 27 at 10:09
12  
@ShadowWizard It can affect any site, it's just a question of scale. Stack Overflow sees this quite often, it's just not as readily noticed. I was originally pushing for a method to allow mods to evict questions from that list (in cache, so it's immediate), but that's not ... easy surgery. I'm going to revisit it after I finish with the MSO/MSE split. –  Tim Post Jan 27 at 10:10
1  
Thanks @Tim, until then you will keep an open eye over the smaller sites? (or of course other CM :)) –  Shadow Wizard Jan 27 at 10:12
    
You call-out community moderation burnout here... which is the only reason I can think of for not advertising your site in this manner. What would be interesting to know is how many users, on average, stuck around and continued to be a contributing member of the community after arriving in this fashion. It could be that the number of new users you gain is more than worth the extra bit of moderation for a few days. –  ben is uǝq backwards Jan 27 at 13:09
    
@benisuǝqbackwards number of (active - visiting) users correlates with number of visits. Visits sharply dropped after "flashes in the pan" => users lost interest –  gnat Jan 27 at 14:03
2  
Lots of related discussion –  enderland Jan 27 at 16:55
2  
@ben, I've updated the question to discuss whether new users brought in are worth the added moderation effort or not by looking at the stats (spoiler: they aren't). –  jmac Jan 28 at 7:54
    
related: What changed in the Hot Questions sidebar algorithm? "...We're testing an alternative method that calculates a score, ranks the questions and then pulls a somewhat larger number off the top, shuffles them, and stuff the top of the deck into the sidebar..." –  gnat Feb 4 at 15:16
    
I don't have an analysis as detailed as this, but I do note that "hot questions" seem to be broader and more subjective than the average good question (on Computer Science) and are certainly not the gems. If hot questions can get us only visitors interested in half-assed content, I'd rather not have them. –  Raphael Feb 7 at 10:02
2  
@ShadowWizard How would omitting beta sites from the hot questions list help expose, promote, and advertise the beta sites, which in the end is what they need in order to graduate? –  Simon André Forsberg Feb 7 at 17:37
1  
great datamining but there are subjective claims mixed in with objective data. overall theres a theme that "we dont want low quality newbies flooding our site". but this is a rather bogus insider perspective. se official policy is more, "how can we convert/recruit these clueless newbies into quality contributors"! that is the key goal, and this analysis largely misses that point entirely! se has found an amazing mechanism to drive newbies to your site, so now how are you gonna train em? –  vzn Feb 11 at 19:58

2 Answers 2

As a mod on UX.stackexchange (a graduated site) we get the same issue. Something hits the HQ top 3 and we get loads of comments, 'answers' and all sorts and invariably the solution is to delete a bunch of posts and protect the question.

So my suggestion is — when a question hits the top 3 of the HQ list then automatically protect it so that <10 rep users can't post directly on that question.

'But we want to encourage people to post, that's the whole point of the HQ list!' I hear you cry. I disagree. We want to encourage people to the site not just to that question. By protecting a question it encourages users to go and post a useful question/answer elsewhere on the site. If they do that, their rep will increase (if they've posted something useful) then they can contribute to the hot question itself. Hopefully by posting a useful question / answer they will then have done a bit more studying into what the site is all about, we'll get extra useful posts and some new users.

Heck, these questions usually get protected anyway, why not just make it automatic?

(I'm aware that there's probably more to it than 'just set it protected automatically'; nothing is ever that simple, but I think it bears consideration at least)

share|improve this answer
3  
'my favorite pattern is from MetaFilter, which is: When we start seeing effects of scale, we shut off the new user page. "Someone mentions us in the press and how great we are? Bye!" That's a way of raising the bar, that's creating a threshold of participation. And anyone who bookmarks that page and says "You know, I really want to be in there; maybe I'll go back later," that's the kind of user MeFi wants to have.' (A Group Is Its Own Worst Enemy) –  gnat Jan 27 at 11:39
    
Not sure that making this automatic is the way to go, because there are many cases where the attention a question gets is actually good and is bringing in new productive users. I'm reluctantly for an 'ejector seat' sort of button for mods to push if things really get out of hand, but showing new users a list of things they can't really touch seems strange (even if only three out of the list). –  Tim Post Jan 27 at 13:54
6  
@TimPost I take your point, but often it's the case that new users come in and post very low quality or duplicate answers and you end up with 20 answers to the question, so I feel the user experience of general readers suffers here as a result and may even put them off revisiting and participating. 'Hey, they accept any old answer on this site'. (This is a hypothesis though). As we invariably have to protect these questions anyway perhaps some sort of trigger can kick in to do it automatically if certain criteria has been met (whatever that is). –  JonW Jan 27 at 13:58
8  
Plus, if it auto-protects that means it's not my name associated to the protection, meaning I don't get called out on Twitter by someone for protecting a question that they had a crappy answer they wanted to post. ;) –  JonW Jan 27 at 14:00
3  
@Tim, I've added data on the new users these questions brought in and their value compared to the harm caused to budding community moderation to my question. (spoiler: they don't seem to be worth it). –  jmac Jan 28 at 7:55

This isn't really an issue of moderation, this keeps happening even on sites where there's at least two mods active at any given time. Questions can stay on that list longer than most people can stay awake - there will always be times where these hit during a gap. I don't think it can be avoided.

I'm not going to go so far as to call this a problem. However, it's something that we should probably address. There's quite a few users on every single site that we have who don't appreciate the party bus driving slowly through their neighborhood with the music up full blast, and they definitely don't appreciate picking up all of the bottles and beer cans after it leaves.

Get on the hot questions list, tweeted and then mentioned on Reddit and things get quite interesting for non-technical sites. On sites where the average person can comprehend every post for the most part - participation does tend to explode.

My proposed solution, which is still very much on the drawing board, allows for moderators or community managers to evict a question from the list provided that:

  • There is definitely evidence of mayhem (as seen by vote/visit, comment/visit, question being protected while featured, and other things)
  • The question has been featured for at least [n] hours (this wasn't just Jeff making fun of a PHP question on Twitter again to get it on the list, and it will soon die down in a sea of 'meh')
  • Other criteria I haven't yet thought of (this is very much just an idea at this point, and the bullets above are also not carved in stone)

The problem is, this is a bit of a technical nightmare. That list is heavily cached because the query that goes into it is very expensive (or at least it was the last time that I heard Jarrod talking about it). Moderators would have to literally be able to evict it right from cache, which presents an interesting problem of a hot questions list that shrinks soon after being compiled.

You're right in expecting help to show up when it's needed - which is part of the reason why the community managers have moderator (and beyond) access on every site. We're also moderators on any site whenever the site needs extra hands. During the week, heck, even half of the weekend we have pretty much 24 hour coverage thanks to time zones.

Doing what you did is precisely what you should have done, and pretty much all you could have done. If your site is having problems and none of your mods happen to be around / awake at the time - any Stack Exchange employee can help you either by jumping in, or locating someone that can. Do not hesitate to reach out to anyone that works here if your site is in trouble and you need help.

Ideal, no - but it can work until something more tenable is put in place.

share|improve this answer
4  
regarding performance "expenses", worth noting that it can be totally avoided if "hotness correction" is limited to posts already picked by current algorithm. Take 100 questions and less than 1000 answers already selected as of now, adjust their score, reorder and feed top 3-to-26 into sidebar. That's it. O(1), performance "cost" is negligible –  gnat Jan 27 at 20:40
3  
Tim, the issue I have in this case is that the only reason I had the wherewithal to go to the meta.so chat is because I'd been there before and figured there was a mod on there. Other users may not be as savvy, and then what? Expecting all community members to know which corner of the ever-growing SE network to visit to get immediate help is going to become less and less practical. At least a quick and easy fix like an @CM ping to call a CM to your chat or something otherwise universal. Maybe a Giant Red Button in chat saying "Help" that lights the CM-signal and calls them over. –  jmac Jan 28 at 5:24
    
...interesting how cut by relative score could turn lemming voting into cure from disease. Passers by tend to vote only 1-2 top answers (short attention span), eventually decreasing amount that contributes to hotness. Take a regular question with answers scored like [5,4,1,0,0 <no cut, all 5 answers "hot">]. Lemmings attack would turn it into something like [30,8, <cut, only 2 answers are "hot"> 1,0,0,0,0,0]. @jmac - would you like this sort of a self-correction? Note btw how it gives a way for community regulars to further fine-tune things by responsible voting on lower scored answers –  gnat Jan 29 at 6:19
1  
@gnat, to be honest I'm starting to think the concept of the hot question is flawed since it looks for popularity (widespread appeal) rather than actually being a good question to understand the site. Yes, reordering the list would help as you suggest, but there has to be an even better way. If we're going to wait to fix this until later, at the very least I'd like to see the community managers around to help support the huge uptick in traffic when questions get up there. –  jmac Jan 29 at 6:25
    
@jmac sort of "quality index"? This is better to discuss with Mysticial, once upon a time he explained that concept to me. As for "hotness" approach being flawed or not, I can't tell for sure, because, you see, so far the list has been operating on the broken implementation of this idea... –  gnat Jan 29 at 6:44
    
...Don't get me wrong, I consider a chance that trying more realistic score will eventually prove that it's just FUBAR... although recent observations of the bottom half of the "big" list make me feel more optimistic about that –  gnat Jan 29 at 6:45
1  
Yeah, a quality index rather than a hot index @gnat. The way it works is it looks for content that will draw eyes, but content that draws eyes usually appeals to the lowest common denominator, and not the experts a site wants to attract. That's why there are so few high-rep users who started from the hot questions. Instead quality should be determined either through special tags by trusted users (applied manually), or by weighting 'hotness' by the voting patterns of top users (rather than all users) to prevent snowballing poor content. And of course using downvotes. Either way is fine. –  jmac Jan 29 at 6:55
    
@jmac In order to draw quality traffic, you'd probably have to look at evergreens, not hotties. Good questions get links, traffic and votes months, even years later. Of course, an influx of crap on these questions would be particularly harmful (auto-protecting sounds good). –  Raphael Feb 7 at 10:06
    
hot list audience consists mostly of SO users, by far the largest site in the network. From this perspective, smaller sites carry the same "exposure risks" as these at SO - but, while SO has 17 moderators, thousands of 20K/10Kers, tens thousands of 3K users with close votes, smaller sites do not have anything like that. To say that such an equality in exposure risks coupled with such a striking difference in abilities to protect from these is unfair, would be an understatement... –  gnat Mar 17 at 13:03
    
...What's more, smaller sites of subjective-ish nature are additionally limited in their abilities to moderate low quality answers because flagging system is designed with primarily technical / coding Q&A model in mind. –  gnat Mar 17 at 13:03
    
here's a recent example of a damage that happens to question attacked by hot list lemmings at smaller / subjective-ish site: workplace.stackexchange.com/q/20683/168 - wonder for how long answers like "tell him you got a huge wiener" could survive in a hot question at SO –  gnat Mar 17 at 14:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .