Here is the current HNQ, as a data point. I can generate an updated version of that at any time if needed.
The major concern I have with jumping to conclusions now is this is just one sample; PPCG could just be having a run lately, and since we're looking for problems (or in a "PPCG again??" mental state), we notice with a bit of confirmation bias. There might be some merit in the "too much weight on answer count" claim (note they're capped at 10, btw), but if that hypothesis is on the right track I think it's more than that: It's sites that attract a lot of high scored answers that gain a lot of weight here. So sites with that activity pattern would presumably tend to appear on the HNQ more. But we'd need to gather data to prove that activity pattern on these sites.
Also, there is a correlation between answer count and question score on average (all sites I checked are like this). Now this considers the average of all questions, so is a wider sample than the HNQ list and therefore may ignore some subtleties of HNQs, but that trend suggests the top half of that equation may increase more than intended, i.e.
AnswerCount are, on average, redundant with eachother. Plus, they're multiplied with each other, so it's a bit of an exponential bias.
All that said, first we should gather samples. A lot of them. Over the course of a few months (some past samples do exist). Table the discussion until then. I'm working on an HNQ tracking app and I'll post it here when I do.
Personally, I don't think that adjusting the core formula is the correct approach. According to How do the "arbitrary hotness points" work on the new Stack Exchange home page?:
Succeeding questions from the same site are penalized by increasing amounts. So, the first question from SO in the list gets multiplied by 1.0, the second by 0.98, the third by 0.96, etc).
And also of course there is:
We make a per-site traffic adjustment so SO does not dominate the entire list.
So if tweaks were to be made I'd say keep the core formula the same, but:
- Increase the successive question penalty, and
- Possibly tweak traffic adjustments, because it could just be that some sites have simply outgrown their current weights.
(Additionally, it might be worth considering making the successive question penalty a per-site parameter rather than a global parameter, so it could be tweaked for certain sites... but I'm not 100% convinced that's the right idea in the big picture sense, it's just a thought.)
By the way, re this comment:
Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ. The hotness formula uses
MIN(AnswerCount, 10) * QScore, so having lots of answers effectively boosts the question score. And the
+ AnswerScore term lets a poorly voted question get by by having good or numerous answers. I think a better formula would also factor in close votes and downvotes directly.
I'm inclined to agree, at least my gut is, but we should consider this carefully, because there's a couple potential counterpoints here. In particular, playing a bit of devil's advocate on purpose just to make sure we're staying sane:
"Too many code golf questions get on HNQ without having many upvotes -- they get them by being on HNQ." -- Well, on the other hand, that's equally true for any question on the HNQ list, not just PPCG. Also, the questions got there without having many upvotes, that is, they got there. So whatever happens afterwards may help to keep them there, but doesn't have impact on them getting there in the first place.
"I think a better formula would also factor in close votes ..." -- Just FYI, close votes are "factored in" in a way: Closed questions are excluded from the HNQ list. So it's not a continuous factor, but as soon as the CV count reaches 5, the question is removed. So there's that.
I do not have enough information to make any conclusions or arguments in any direction, my intent is just to make sure we're considering this carefully.