Last week we completed the first 100 days since the HNQ changes went into effect. While we've made some other changes since then (e.g. minimum question age of eight hours), I figured that now is a good time to look at the numbers and see how this change has impacted the network. I'd like to look at them again when we hit six months or a year because, well... the HNQ is a fickle beast.
Right now, many sites look like they had huge drops or surges in their presence on the list but when you're sampling short time periods and the difference is one question either way because it's a low-volume site, it's going to look extreme. So, when you look at percentages, it's important to also look at the question count.
That said, we don't have a question count in SEDE from before this update due to not having a post history event that helped us track HNQs (for the pedantic, yes, that change went out a few days before but not long enough to be as useful here). So... we're kinda hampered in that but I'll get to this more later.
What I can look at in SEDE is the number of link clicks from the HNQ list and what percentage of total site directions those clicks are. The comparisons below are between the 100 days immediately before and after the changes to the HNQ list on March 11, 2019.
The top twenty sites by total HNQ traffic clicks before the update (fifth column):
% HNQ % HNQ CLICKS CLICKS
SITE NAME BEFORE AFTER CHANGE BEFORE AFTER CHANGE
The Workplace 17.7% 15.3% -13.3% 1568566 1161275 -26.0%
Worldbuilding 19.2% 15.6% -18.8% 657930 505964 -23.1%
Academia 10.5% 10.1% -4.1% 638862 529342 -17.1%
Sci-Fi & Fantasy 3.6% 4.8% 32.8% 475335 639892 34.6%
RPG 3.3% 2.7% -16.5% 440328 363327 -17.5%
Travel 4.0% 4.1% 1.1% 395468 395603 0.0%
Info Security 3.7% 3.0% -19.3% 365383 306157 -16.2%
Politics 14.0% 15.3% 8.8% 313273 346610 10.6%
Money 7.3% 8.8% 21.2% 303044 379524 25.2%
Stack Overflow 0.0% 0.0% -6.7% 234866 224201 -4.5%
Physics 1.7% 2.0% 16.2% 233108 243694 4.5%
English Language & Usage 0.5% 0.4% -21.6% 232371 164796 -29.1%
Mathematics 0.4% 0.3% -33.6% 229613 141442 -38.4%
Writing 13.8% 14.2% 3.3% 223687 235209 5.2%
Skeptics 11.3% 9.2% -17.9% 206766 142507 -31.1%
Movies & TV 2.2% 4.3% 94.3% 201073 440532 119.1%
Puzzling 8.5% 9.5% 11.6% 187167 187373 0.1%
Space Exploration 11.2% 11.0% -1.7% 169263 158941 -6.1%
English Language Learners 1.0% 1.1% 12.6% 168525 216138 28.3%
Aviation 3.9% 4.9% 26.4% 165470 251357 51.9%
Changes in traffic (clicks)
At the very top there are some big reductions, likely due to the cap of five questions at a time per site but as you move down things bounce around so that, within the top twenty, there's ten sites with fewer clicks, two about the same and eight that increase. Movies & TV, which more than doubled its views, and Sci-Fi largely saw a boost due to the release of several blockbuster films, particularly Avengers: Endgame (April 26, 2019).
On the other end, the site with the biggest reduction in clicks percentage and traffic is also a site that gets less than 1% of its traffic from the HNQ list - Mathematics. What's so interesting to me here is that Math has the highest number of questions on the HNQ list, so the clicks/question is quite low.
How has this impacted sites that get much of their traffic from the HNQ list?
I really love seeing which sites get the highest percentage of their clicks from the HNQ and, actually, the winner there both before and after the update is Retrocomputing, with 19.4% of traffic coming from the HNQ before the change and 23.7% after. They're not in the top 20 by total clicks as they have only ~100k before and ~150k after.
While Retrocomputing saw an increase, four of the ten sites that had >10% of their traffic from the HNQ list before the change saw a decrease of a few percentage points, particularly Computer Science Educators, which dropped from 15.4% to 10.5%, a loss of about 30% - but from the same quantity of questions - six. The total change in clicks is about 1200.
How many questions are we talking about?
Did these changes, for example, lead to differences in the number of questions on a site that hit HNQ
I mentioned before that I can't really get you SEDE data for this prior to the update. Fortunately, the marvelous Glorfindel had started tracking the HNQ list in August 2018, prior to these changes going into effect, so I do have some data, which he's shared with me. So, combining the two I get the following info:
More questions from the same number of sites
- Before: 13,578 questions featured on 154 sites
- After: 14,410 questions from 154 sites
This is an increase of 832 questions or 6.13%. This seems like a nice improvement! I'm guessing that a big part of this is maximum of three-days on the list we added.
How did question count change on a per-site basis?
Of the 170 eligible sites, 160 had at least one question on the HNQ list during the 200 days, which means 10 sites didn't appear on the list at all.
Of the 160 sites,
- 76 sites saw an increase of two or more questions on the HNQ list,
- 34 saw about the same amount of questions (change of -1 to 1)
- 50 saw a reduction of two or more questions
The 160 sites individually had the following number of posts before and after the update - note that the buckets are a binary progression starting at 2 (because who doesn't like a little binary... plus the groups tend to squish together at the bottom):
Pre-HNQ Post-HNQ
Change Change
<2 questions 17 22
3-4 questions 13 8
5-8 questions 15 11
9-16 questions 17 18
17-32 questions 21 19
33-64 questions 19 20
65-128 questions 23 21
129-256 questions 14 20
257-512 questions 10 11
>513 questions 5 4
Which I made into a pretty chart:
For this time period, there's not a huge change in the quantity of sites with a certain number of questions that hit the HNQ. The line's a bit smoother but no drastic changes.
What does it all mean?
I've just showed you a bunch of numbers and they... don't really seem to show a huge overall change.
- increase in the number of questions seen (YAY!)
- more sites saw an increase in total questions than a reduction (YAY!)
- same (pretty high) number of sites are getting featured (Meh)
There's a lot of other factors that we still can't really take into account to either truly quantify the effects but it's also difficult to qualify the impact (are the questions on the HNQ list better questions?).
- numbers may be confounded by external factors/events we can't control
- we don't know how long questions stay on the HNQ list (we don't measure it)
- we don't know if the questions are generally better ambassadors for a site or more representative of the site quality.
Before I close, I want to talk a bit about the primary goals of the HNQ list update. The big ask was to help manage it a bit in a few key ways. Many of the features we implemented were designed to give control to moderators and users that we'd failed to implement and that was causing problems but also to add some tracking features. Additionally we set new limits on questions/site at a time and how long a question could stay on the list.
Moderator removal tool
While most sites haven't used this feature, some use it frequently. During these first 100 days 321 questions were removed from the HNQ list, mostly on the Mathematics site. Math seems to have the most questions hit the HNQ both before and after the update (1031 and 1010, respectively) but of the 1010 added during the first 100 days, 181 (17.9%) were removed by mods.
Site setting to remove the HNQ from the right sidebar
Is there a number of people who have used their profile settings to turn off HNQ entirely and is there any decline in traffic as a consequence?
Right now there's a bit over 1300 people using the "hide HNQ" setting. Not a huge amount in the grand scale of Stack Overflow but not bad for ~110 days. Comparing the total number of HNQ clicks before and after gives a gain of 211 clicks (10,201,659 before and 10,201,870 after). So... there doesn't seem to be a change in total clicks at all.
Conclusions
There's some ups and downs but, overall, I think the change has been a positive one both in the quantifiable changes and control of the HNQ list content. Even still, when reflecting on what the goals of the update were, I would call this a success.
