14

Elsequestion it was said:

Guess what: if your site is full of crappy questions, your site sucks - even if they're not highly-ranked by your own users, folks are finding them via Google, and that's where the vast majority of your readers are coming from. You can work to fix that - as painful as that process is - or you can bury your head in the sand and blame it on all of those stupid people from elsewhere. If you think "hot" questions are a serious problem for your site's quality, then you're already ignoring a much bigger problem. Blaming someone else is easy and fun for the whole family - but it doesn't fix anything.

This struck me as a bit odd and I'd really like to see some numbers to back it up, that the active community on the site is the one pushing the questions that are popular and being raised as issues as to why hot questions are problematic.

It's one thing if it's the core group on the site that is pushing these questions into troublesome popularity, it's quite another if it's people who are less involved in the community and the core group lacks the ability to keep it from growing too popular.

It would be really nice to get a "how many of the up votes came from people who hit the question via a hot question rather than from google". I am not sure the problem is folks finding them from Google... and without the ability to find the questions that are outliers from anonymous and low rep feedback (typically from Google?) it becomes difficult to identify the question people are finding from the Google that should be protected.

I would like to see some stats that are broken down by:

  • Reputation
  • Answers given (i.e. out of "20 votes, 15 of them were from people who have answered 0 questions")
  • Days active out of the last 100 days

For the following questions that are ones that are currently or have been outlierly popular:

(I'm trying to get a sampling of sites where I recall the question being 'hot' and relatively recent so the data is 'fresh')

To an extent what needs to be distinguished and identified is "does the core group within the site have the tools / ability to be able to clean its site up or protect itself from the added popularity these questions bring?". I hope that the numbers from the above could give some added data and insight into this. Do we need to protect things faster? Would The association bonus should not enable users to vote on every site help reduce the extreme scores?

For context on the core group, please read A Group Is Its Own Worst Enemy by Clay Shirky. In particular pay attention to "Three Things to Accept" #2 and #3, and "Four Things to Design For" #3.

Members are different than users. A pattern will arise in which there is some group of users that cares more than average about the integrity and success of the group as a whole. And that becomes your core group...

The core group has rights that trump individual rights in some situations... you can see examples of how bad an idea voting is when citizenship is the same as ability to log in.

...you need barriers to participation. This is one of the things that killed Usenet. You have to have some cost to either join or participate... Now, this pulls against the cardinal virtue of ease of use. But ease of use is wrong. Ease of use is the wrong way to look at the situation, because you've got the Necker cube flipped in the wrong direction. The user of social software is the group, not the individual.

|improve this question
  • 1
    I'll see what I can pull together here. Fair warning: I don't expect answers to be interesting at all; a pretty big chunk of all votes come from folks who've never posted any answers, even on Stack Overflow. – Shog9 Feb 8 '14 at 23:23
  • 3
    @Shog9 for clarification - its answers on that site. The bit I'm wondering about is "is this from the people you are trying to attract to the site" - the experts, the ones who contribute the "A" part of Q&A. If I go and up vote hot questions on TeX or Space even though I've got a few answers elsewhere I'm not one of the people that you're trying to attract there because I came by to up vote the pretty pictures. The key to the survival of an SE site is attracting experts in the field - not people looking at pretty pictures. – user213963 Feb 8 '14 at 23:36
  • 1
    Hmm... Now I'm kinda confused on what you're trying to prove from this. It's easy to separate votes from folks off-site and those on-site, or even folks who're participating on a site and those who aren't. Separating "expert" votes from "non-expert" votes is another matter. – Shog9 Feb 8 '14 at 23:57
  • 2
    @Shog9 A number of sites have recognized (either raised the barrier to entry or folded) because of the difficulty of attracting people who provide answers to the questions and stick around (generally, these are the same). What I am curious about is are the people who are coming by and voting on hot questions part of the community that the site has established, or not. Who is voting on these questions? who is making them popular? is there something that the core community on the site can do to abate the unnatural popularity of the questions? – user213963 Feb 9 '14 at 0:51
  • Ok - that's what I initially assumed you were going for, so that's what I tried to answer. I'm afraid the conclusions you can draw from this may be limited simply due to the overwhelming popularity of the questions you selected; if you'd picked more controversial / less universally-loved examples, the results may have been more interesting. – Shog9 Feb 9 '14 at 0:56
  • 3
    ... Are we to blame for them getting meh questions popular in the first place? There is a bigger picture to this - its fairly consistent that the smaller sites have difficulty with even a fraction of the attention that SO can focus on any of our sites - is this coming from elsewhere on the SE network or is it coming from within our own community? Without some numbers (and I am doing a shotgun approach to the data) we (the people who actively review, close, delete at the various sites) are blind to how to address and prevent the eye of SO from the meh questions. – user213963 Feb 9 '14 at 0:56
  • 2
    are you looking for voting stats on question and answers or on question only? I ask because it looks like that the answer given to you appears to give stats only for questions – gnat Feb 9 '14 at 11:36
  • FWIW, I was rather surprised by the stats on votes from "previously-posted-answer" users - if anything, answerers are over-represented in the voters on these questions. – Shog9 Feb 9 '14 at 17:49
  • 1
    @Shog9 could we also get the information on the top answer for each question in that set? While questions are helpful to get a picture, but from my understanding of the hotness score, it is the votes on the answers that drive them to the disproportionate extremes of popularity. It may also be a useful to do a retrospective in a month or so and look at these questions - the people who had just the association bonus or feedback from registered but low rep users and see if they are remaining to be part o the community or if it was more of a drive by buzzfeed type thing. – user213963 Feb 9 '14 at 21:15
  • @MichaelT: I think you're being a bit myopic here - these are interesting questions, but when we're working with such a small sample it's hard to draw any good conclusions. Why not ask about the behaviors of new/off-site voters in general? – Shog9 Feb 14 '14 at 17:26
  • 1
    Its a starting point to think about what data there is and what direction future investigations should go. This is in the context of how much hot questions help an SE community grow (or not). It would be very interesting to get some data in a month or so on the people who voted on these questions and compare their before and after voting activity. – user213963 Feb 14 '14 at 17:30
  • @Shog9 can we get data similar to one quoted here but closer focused on hot questions, that is limited to ones with over 1-2K views and including deleted posts? – gnat May 6 '14 at 4:54
  • 1
    You should post something on TWP's meta if you want more data for that site, @gnat - I'll try to dig this up tomorrow. – Shog9 May 6 '14 at 5:33
  • @Shog9 there you go - Request for hot question vote statistics at TWP meta – gnat May 6 '14 at 15:17
  • related: Does Stack Exchange record when particular question enters and leaves hot list? – gnat Aug 22 '14 at 17:19
10

First off, this made me laugh:

This struck me as a bit odd and I'd really like to see some numbers to back it up

You quoted what I intended to be a tautology, an assertion that a site full of crappy, popular questions is a crappy site. But I guess we don't talk about search engine traffic that much; for reference, it makes up 80-90%+ of all traffic to all of the sites you referenced: chances are, if someone's viewing your question it's someone who found it via search. Hot questions are dramatic because they get a disproportionate amount of on-site and in-network traffic, visitors who can visibly affect the question and answers they're viewing - but this is a drop in the bucket of folks silently reading.

That out of the way, the association bonus is a common point of concern: the notion that hoards of folks unfamiliar with your site could descend upon it and skew the voting without ever having earned the privilege is worrying, and rightly so - indeed, it was one of the big concerns for the folks over on MathOverflow after they joined the network. But in practice, it's rarely an issue; for popular questions, new folks tend to vote the same way that folks already on the site vote. Indeed, the real skew shows up with obscure questions and answers that only get one or two votes total.

But talk is cheap. Below is the data for the questions you listed. Note that all voter classifications are relative to the time when the vote was cast - in other words, a voter whose first action on the site was to cast the vote but who later posted several answers, earned 15+ reputation points, etc. isn't counted as an established user in those categories.

How are pseudorandom and truly random numbers different and why does it matter?

  • Total Upvotes: 340
  • Total Downvotes: 2
  • Upvotes from users with vote privilege earned on site: 176
  • Upvotes from users with assoc bonus only: 115
  • Upvotes from users with at least one prior answer on site: 182
  • Upvotes from users with at least one day visited in the week prior to voting: 188
  • Anonymous upvotes: 38

What is a 'friendly' way to let managers know that having good developers is a privilege?

  • Total Upvotes: 198
  • Total Downvotes: 5
  • Upvotes from users with vote privilege earned on site: 58
  • Upvotes from users with assoc bonus only: 130
  • Upvotes from users with at least one prior answer on site: 45
  • Upvotes from users with at least one day visited in the week prior to voting: 59
  • Anonymous upvotes: 119

How has an increase in the complexity of systems affected successive generations of programmers?

  • Total Upvotes: 109
  • Total Downvotes: 1
  • Upvotes from users with vote privilege earned on site: 66
  • Upvotes from users with assoc bonus only: 29
  • Upvotes from users with at least one prior answer on site: 68
  • Upvotes from users with at least one day visited in the week prior to voting: 74
  • Anonymous upvotes: 33

Would the One Ring even work for anyone but Sauron?

  • Total Upvotes: 170
  • Total Downvotes: 0
  • Upvotes from users with vote privilege earned on site: 69
  • Upvotes from users with assoc bonus only: 94
  • Upvotes from users with at least one prior answer on site: 61
  • Upvotes from users with at least one day visited in the week prior to voting: 97
  • Anonymous upvotes: 105

Is there a non-sexual phrase for sleeping with someone?

  • Total Upvotes: 176
  • Total Downvotes: 3
  • Upvotes from users with vote privilege earned on site: 79
  • Upvotes from users with assoc bonus only: 76
  • Upvotes from users with at least one prior answer on site: 56
  • Upvotes from users with at least one day visited in the week prior to voting: 67
  • Anonymous upvotes: 59

Why is "except: pass" a bad programming practice?

  • Total Upvotes: 116
  • Total Downvotes: 3
  • Upvotes from users with vote privilege earned on site: 112
  • Upvotes from users with assoc bonus only: 1
  • Upvotes from users with at least one prior answer on site: 111
  • Upvotes from users with at least one day visited in the week prior to voting: 113
  • Anonymous upvotes: 0
|improve this answer
  • 1
    "But in practice, it's rarely an issue" On MO one (or at least some there) was well aware it would only be an issue rarely, but it got confirmed it is an issue sometimes (not a huge one but still it is an issue). While the actual upside there is to voting-by-association still remains quite unclear. – quid Feb 9 '14 at 11:59
  • 1
    missing stats on answers voting make this data sort of self-fulfilling prophecy , in the same way Jeff and Joel were worrying about 6 years ago. Get an average but reasonable question, force it to the top of collider / sidebar and it will start collecting upvotes (what else, as it's reasonable) not because it's that good but because it is forced to be exposed... – gnat Feb 9 '14 at 12:10
  • 1
    ...Ignoring voting on answers (both in formula and in stats) makes any answer (no matter how crappy it is evaluated by "core group") bump and further increase question exposure - again, so that question collects upvotes (what else, as it's reasonable) – gnat Feb 9 '14 at 12:11
  • 3
    To say something more positive: what I do however now see differently than before (due to some other data) is that the hot list itself is really not that much of a problem. Since its effect seems rather negligible relative to thing being posted and successfull on reddit or something along these lines. Yet voting-by-association still remains some problem, for some sites at least. But thanks for the data (especially the earlier one on MO)! – quid Feb 9 '14 at 13:24
0

There is more detailed data for particular site (Workplace) provided by Shog when asked on hot-question traffic here. Data is for 180 days up to 2014-03-27:

Defining "outsider votes" as votes cast by someone who hasn't yet earned even 15 reputation on the site at the time the vote was cast

And looking at all votes cast during the past 180 days

We'll define "outsider agreement" as a case where outsider votes on a post are > 0 and insider (folks who've earned >= 15 rep on the site) upvotes - insider downvotes are also > 0

We'll define "outsider disagreement" as a case where outsider votes on a post are > 0 and insider votes are < 0

Cases of outsider agreement: 2,680

Cases of outsider disagreement: 277

Cases where the insiders' votes would've brought a post's score < 0, but outsider votes were present in sufficient quantity to make the post's score positive: 147

Total posts voted on by outsiders in the past 180 days: 3870

Total posts voted on by insiders in the past 180 days: 7186

Total posts voted on by both groups in the past 180 days: 3053

Q: What is the difference between outsider disagreement (277) and insider negative but for outside votes (147)?

A: So... Imagine you & I see an answer & we both downvote it. It's at -2. Then 3 people come in with nothing but the association bonus and up-vote it. It's at +1. There are 147 cases like that

Now imagine @enderland and @RhysW both downvote the same answer, so it's at -1 again. There are 277 cases like that.

Total insider votes during the past 180 days: 29,479

Total outsider votes during the past 180 days: 12,837

Q: am I reading this right -- there were ~700 posts that only outsiders voted on (insiders didn't)?

A: something like that

I'm not breaking this down any further right now, but a really common case for that involves answers to questions being upvoted by the asker

Q: Ok, so if I'm reading all this right, we have significant "outside" participation, but only in 5-10% of cases does that participation disagree with the community.

A: Right. And if you consider that even most of those 5-10% are "controversial" in the sense that they're getting both up-votes and down-votes from insiders, you can see there's a lot to be gained from community education.

Two notes regarding above data. First, it is complete in the sense that it takes deleted posts into account 1.

Second, it doesn't seem to distinguish between questions that were in hot list and those that weren't. As a result, one can only (maybe erroneously) assume that most "outsider" votes in above relate to hot questions. Probably taking data only for questions having over 500-1000 views could make a closer approximation.

|improve this answer
  • 1
    Generally, unless I specifically indicate that I'm ignoring deleted posts I'm including them in these things. This data definitely included voting on deleted posts. – Shog9 May 6 '14 at 4:56
  • thanks @Shog9! I updated the post with this clarification – gnat May 6 '14 at 5:02
  • Re-reading this, I'm realizing that limiting this to 180 days worth of voting data (without also limiting the posts being voted on to that range) made this kinda sketchy - depending on the timing, this could easily skew the votes on a given post significantly. – Shog9 May 6 '14 at 5:32

You must log in to answer this question.