36

I was recently browsing through the Hot Network Questions and spotting a weird pattern with the frequency of the sites. So I decided to tally up the results and what I found was not reassuring. These are the sites with more than two questions in the Hot Network Questions and the total questions they had:

The bottom of this seems OK but the top 3 are a little alarming. Especially the Code Golf one. I think that you can agree that 11 in the top questions is a little much.

Some of you will surely comment with "Clearly they have a lot of top quality questions". But that isn't true. Or at least it isn't the reason that they have this many. The reason is quite simple: code golf is a competition and so it sure to attract more answers and views than a regular question.

Therefore, I am suggesting that we change the way that the HNQ bot checks for questions. The bot should only promote, at maximum, 5 questions to the HNQ list. I have seen this pattern happen week after week, where code golf takes at least 10 of the HNQs. If we change the algorithm, more questions from less populous sites will be able to gain recognition, allowing them to gain more users and thus expand.

If a site has reached the 5 questions and another question becomes incredibly popular, then the lowest ranking HNQ from that site is removed and the new one replaces it.

Or, for an alternative option, see Catija's suggestion

What do you think? Any questions?

  • 4
    Or, rather than a hard limit, just ramp up the difficulty exponentially after 3-4 questions being on HNQ. – Catija May 4 '17 at 19:39
  • 10
    For reference, the actual algorithm is available here: meta.stackexchange.com/questions/60756/… There's definitely a debuff for additional questions from the same site but I think that there could easily be a bigger one. Some sites just have lots of active voters and others do not. Voting is such a huge part of the algorithm. – Catija May 4 '17 at 19:42
  • 4
    I'd prefer if the solution addressed the underlying problem that HNQ heavily favors high-activity questions, so that when code golf questions do get featured, it's the best ones rather than the easiest ones. – xnor May 5 '17 at 2:24
  • 1
    @xnor So... you want a computer to parse question content and "know" somehow that the questions are "good"? Voting is how we're supposed to indicate a good question... if people choose to upvote easy questions instead of good ones, that's a voting problem, not a qualification problem. – Catija May 5 '17 at 16:25
  • 7
    @Catija I'm not looking for magic, just better use of the stats. Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ. The hotness formula uses MIN(AnswerCount, 10) * QScore, so having lots of answers effectively boosts the question score. And the + AnswerScore term lets a poorly voted question get by by having good or numerous answers. I think a better formula would also factor in close votes and downvotes directly. – xnor May 5 '17 at 18:37
  • 2
    @user00001 As I understand, downvotes are accounted for in QScore, but I think they send a stronger signal than cancelling an upvote and so should carry a greater penalty in the HNQ formula. – xnor May 5 '17 at 18:59
  • The algorithm incorporates per-site adjustments based on traffic. The algorithm could probably stay the same, but perhaps it's time to tweak the weight of certain sites. Still, I recommend watching this for a longer period of time, one sample isn't really enough to show an ongoing pattern or preference. – Jason C May 6 '17 at 22:19
  • 1
    This problem would also be mitigated by solving the much bigger problem that HNQ just favours questions with a ton of answers (disregarding individual answer quality). – Chris says Reinstate Monica May 7 '17 at 1:09
  • The top three now are "codegolf.stackexchange.com (8), worldbuilding.stackexchange.com (4), english.stackexchange.com (4)" and the numbers have shifted a lot. Stressing again the need to watch this for a while (like, a month) and the low value of a single sample. – Jason C May 7 '17 at 4:04
  • @JasonC FWIW collection of daily samples for about 4 months between Dec 2016 and Apr 2017 can be obtained from web archive as discussed here – gnat May 7 '17 at 6:45
  • I think there is already a per-site "weight" that CM's can change. – Shadow The Princess Wizard May 7 '17 at 10:35
  • @ShadowWizard There is. There's also a global decay parameter for successive questions from the same site, which could probably be either increased, or made into a per-site parameter. – Jason C May 7 '17 at 14:23
  • 2
    While I still haven't gotten around to gathering data yet, I have to say, I'm pretty confident ppcg is getting a bit too much spotlight. I kind of feel like it should just not be allowed to be on the list at all. The hnq is good for learning about new things but ppcg isnt really good for opening people up to new topics. It's just a game site. – Jason C Jun 14 '17 at 14:57
  • related: Trolls in our Halls (@ShadowWizard do you remember this one?) – gnat Jun 15 '17 at 21:22
  • @gnat lol of course, though the incentive for my own request was the trolling titles, usually I don't really care about this. – Shadow The Princess Wizard Jun 16 '17 at 5:48
16
+50

The biggest problem seems to be that the number of answers has such a big effect, no matter what the answer quality is. Deleting spam answers from a question should not drop its hotness, but now it does. In my opinion answers with zero score have a disproportionate effect on hotness.

This same mechanism is what promotes the code golf questions too much according to my observations. I think that the weight given to the number of answers is a significant cause of imbalance between the sites. Fixing this should improve the situation, but I have no hard data to back this up.

The original hotness formula (see here) contains

(MIN(AnswerCount, 10) * QScore) / 5 + AnswerScore

where the weight due to answer count is clear. I think something like

QScore * SUM(IndividualAnswerScore * Weight(AnswerNumber))

would work better. (Or maybe the first multiplication could be replaced with addition?) Here Weight(k) is some weight, for example Weight(k)=1/k. So if the question has score 10 and its three answers have scores 15, 6, and 3, the total score would be 10*(15/1+6/2+3/3)=190.

By choosing the weight differently you prioritize different shapes of answer score distribution. If the weight decays rapidly, you only look at the first few answers. If the weight is constant, you just sum up the answer scores. If the weight is constant for k<4 and zero from k=4 onwards, then you sum the scores of the first three answers. I prefer to have some decay; one answer of 10 points is worth more than ten answers with one point each. Having more upvoted answers increases hotness as before — with the exception of cutoffs when the weight is zero from some point onward.

Or, for a simpler alternative, drop the answer cap from 10 to 3 or something in the formula. To some extent this can be realized with suitable weights as mentioned above. I don't see the relevance of the number of answers after three or so. But I do prefer the weighted sum if feasible.

  • "Deleting spam answers from a question should not drop its hotness." Should not or does not? – Wildcard May 9 '17 at 22:21
  • 2
    @Wildcard It should not affect the hotness formula to delete posts with zero (or slightly negative) score. Currently having more answers greatly increases the hotness given by the formula, even if they have slightly negative score. As a moderator I can help questions stay on the HNQ list by letting horrible answers stay there, while the quality of the whole thread would improve by deleting them. This is one of the problems I see with the current system. – Joonas Ilmavirta May 10 '17 at 4:38
  • 3
    What an odd view for a moderator to take. I find that many moderators (especially on Workplace and Academia) would like to get questions off the HNQ. But thank you for clarifying. – Wildcard May 10 '17 at 4:53
  • 2
    @Wildcard I meant that I'm able to do so. I'll clean up any unwelcome content as soon as I can. My site is Latin, and the small site can use any visibility it can get, but I don't want to promote it by letting spam stay. I would much like to be unable to promote it that way. – Joonas Ilmavirta May 10 '17 at 5:01
  • @JoonasIlmavirta you might be interested in related discussion at CB.SE: How can I get people to join a site and not simply glance and pass it by? "First thing to keep in mind is that purpose of hot questions feature is established pretty firmly and that it's not really about helping in your community growth... Because of this, you should not generally expect this feature to work in favor of your community. You can refer examples at Programmers and Workplace to find more details about that..." – gnat May 11 '17 at 8:41
  • This doesn't even address the main point of the question - there are too many questions from one or two specific sites on the HNQ. – Catija Jun 6 '17 at 16:15
  • 4
    @Catija I should have made that clearer in my answer. See the second paragraph. I think the weight for answer number is a cause of imbalance between sites. Sites have greatly different characteristic answer numbers per question. – Joonas Ilmavirta Jun 6 '17 at 16:20
9
+50

Here is the current HNQ, as a data point. I can generate an updated version of that at any time if needed.

The major concern I have with jumping to conclusions now is this is just one sample; PPCG could just be having a run lately, and since we're looking for problems (or in a "PPCG again??" mental state), we notice with a bit of confirmation bias. There might be some merit in the "too much weight on answer count" claim (note they're capped at 10, btw), but if that hypothesis is on the right track I think it's more than that: It's sites that attract a lot of high scored answers that gain a lot of weight here. So sites with that activity pattern would presumably tend to appear on the HNQ more. But we'd need to gather data to prove that activity pattern on these sites.

Also, there is a correlation between answer count and question score on average (all sites I checked are like this). Now this considers the average of all questions, so is a wider sample than the HNQ list and therefore may ignore some subtleties of HNQs, but that trend suggests the top half of that equation may increase more than intended, i.e. QScore and AnswerCount are, on average, redundant with eachother. Plus, they're multiplied with each other, so it's a bit of an exponential bias.


All that said, first we should gather samples. A lot of them. Over the course of a few months (some past samples do exist). Table the discussion until then. I'm working on an HNQ tracking app and I'll post it here when I do.

Personally, I don't think that adjusting the core formula is the correct approach. According to How do the "arbitrary hotness points" work on the new Stack Exchange home page?:

Succeeding questions from the same site are penalized by increasing amounts. So, the first question from SO in the list gets multiplied by 1.0, the second by 0.98, the third by 0.96, etc).

And also of course there is:

We make a per-site traffic adjustment so SO does not dominate the entire list.

So if tweaks were to be made I'd say keep the core formula the same, but:

  • Increase the successive question penalty, and
  • Possibly tweak traffic adjustments, because it could just be that some sites have simply outgrown their current weights.

(Additionally, it might be worth considering making the successive question penalty a per-site parameter rather than a global parameter, so it could be tweaked for certain sites... but I'm not 100% convinced that's the right idea in the big picture sense, it's just a thought.)


By the way, re this comment:

Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ. The hotness formula uses MIN(AnswerCount, 10) * QScore, so having lots of answers effectively boosts the question score. And the + AnswerScore term lets a poorly voted question get by by having good or numerous answers. I think a better formula would also factor in close votes and downvotes directly.

I'm inclined to agree, at least my gut is, but we should consider this carefully, because there's a couple potential counterpoints here. In particular, playing a bit of devil's advocate on purpose just to make sure we're staying sane:

  • "Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ." -- Well, on the other hand, that's equally true for any question on the HNQ list, not just PPCG. Also, the questions got there without having many upvotes, that is, they got there. So whatever happens afterwards may help to keep them there, but doesn't have impact on them getting there in the first place.

  • "I think a better formula would also factor in close votes ..." -- Just FYI, close votes are "factored in" in a way: Closed questions are excluded from the HNQ list. So it's not a continuous factor, but as soon as the CV count reaches 5, the question is removed. So there's that.

I do not have enough information to make any conclusions or arguments in any direction, my intent is just to make sure we're considering this carefully.

  • 5
    "Increase the successive question penalty, and possibly tweak traffic adjustments, because it could just be that some sites have simply outgrown their current weights." Which I could upvote more than once for this. Conservative, and almost certainly the correct solution. – Wildcard May 9 '17 at 22:24
  • Dynamic adjustment based on e.g. the last three months' mean traffic. Discount successive questions using a factor of e^{-kx}, x is the number of questions from the site already on the list and k is some value supported by testing to establish a "good" rate, so that the first question gets all the score it deserves but later questions get knocked increasingly harder. – Nij Jun 5 '17 at 19:37
  • I’m taking an HTML snapshot of the SE homepage every hour. This gives us the top 30 questions. I’ll look into getting all 100. It’s been going for 80 hours - I’ll keep it running. – Tim Jun 9 '17 at 0:20
  • data avaliable at 79736f7e78.dataplicity.io/ImageHosting/hotquestions.csv – Tim Jun 9 '17 at 12:48
  • While I still haven't gotten around to gathering data yet, I have to say, I'm pretty confident ppcg is getting a bit too much spotlight. I kind of am starting to feel like it should just not be allowed to be on the list at all. The hnq is good for learning about new things but ppcg isnt really good for opening people up to new topics. It's just a game site. – Jason C Jun 14 '17 at 14:59
  • @JasonC I've collected data for the top 30 HNQs for the past 208 hours - since 00:00 2017-06-06 (over 1 week now). In that time, I've seen an average of 3.69 Code Golf questions (8.12% of the front page). There has always been at least 2 in the list (6.67%), and at one point there were 8 (2017-06-08 21:00:00 UTC) (26.67%). – Tim Jun 14 '17 at 16:55
  • @JasonC Here's the graph: imgur.com/a/iQzi0 – Tim Jun 14 '17 at 19:27
  • 2
    a thing to keep in mind is that purpose of hot questions feature is established pretty firmly and that it's not really learning about new things. The feature is there only to show network wide audience entertaining / interesting questions. If you're interested in more details on that refer What is the Goal of “Hot Network Questions”? – gnat Jun 14 '17 at 21:24
  • 2
    ...if you want to justify changes based on official purpose of HNQ the only way to make a point seem to be to somehow prove that current way isn't entertaining (eg low on views). Something like that was done in the past when few Workplace regulars made apparently, offensively boring PHP question stick at the top of the list for almost a week. Soon after that SE team introduced shuffling in the list – gnat Jun 14 '17 at 21:34
  • @gnat That's true. Well in any case, I left it as a comment because I definitely don't have a concrete case based on anything, and my comment is not a good argument for a change. I'll check out Tim's data too, I'm on vacation right now, though, beach trumps meta. :D – Jason C Jun 14 '17 at 21:52
4

As of March 11th, 2019,

Each site can only have a max of five questions on the HNQ list at any given time.

This is a big, much-requested change and we may reduce the number in the future even further. We're starting it higher than some might want (suggestions went as low as one per site) because sites that have had a lot of exposure through the HNQ may see a dramatic drop in visits, so we need to be careful to find the right number here and possibly do some testing at different levels.

This number is also configurable on a per-site basis, so if a site wants to reduce their HNQ footprint, we can lower it even more, even to zero if a site wishes to be excluded entirely. Sites will needs to go through a meta discussion before requesting this change and it will be up to the site itself to request a change rather than having the limit imposed upon it (unless we lower the maximum for the entire network). So for example, Stack Overflow can't vote to kick Movies & TV off the list entirely because they don't want to see spoilers for the last season of Game of Thrones, but Movies & TV can ask that fewer of their questions be shown so that they can devote sufficient time to those that are.

2

Here is the data I've collected. This updates every hour. Note that times are in BST (GMT+1).

It starts at 00:00 2017-06-06 GMT. As of writing this post, it has data for 210 hours (one hour the internet connection failed).

Here's a graph of the number of Code Golf questions on the front page of stackexchange.com (the top 30 HNQs):

Blue is the number for any given hour. Red is the average for all time and yellow is the rolling average for the past 15 hours.

Feel free to analyse the data in there - there's not much right now, but it's getting there.

  • I'd be curious to know if it's possible to include both pages... I did a count this morning and there were 11 total between the two. :/ – Catija Jun 14 '17 at 21:42
  • @Catija this seems to be a little bias towards saying CG has more than they do (as a percentage). Including both is possible, but harder than I expected and I'm very busy for the rest of the month. – Tim Jun 14 '17 at 21:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .