30

It's been almost two months since some major changes were made to the Hot Network Questions (HNQ), including a timeline event when a post hits HNQ, a (configurable) maximum number of questions each site can have at a time in HNQ, posts ageing out of HNQ and moderators being able to remove questions from HNQ.

The last paragraph of that announcement states:

We know that there's still work to do but these changes will address the most immediate needs and give us the information we need to make changes in the future that are based on data rather than anecdotal evidence.

I'm wondering if some of this data has already been collected, and can be shared. Are these changes are already having some noticeable effect; have they actually changed anything since being implemented? Did these changes, for example, lead to differences in the number of questions on a site that hit HNQ, the time these questions spend on HNQ, site traffic, or perhaps even post scores? Is there a number of people who have used their profile settings to turn off HNQ entirely, and is there any decline in traffic as a consequence?

  • For a recent update of the changes see: meta.stackexchange.com/a/327213/282094 --- For an answer which is pretty close to a duplicate: meta.stackexchange.com/questions/326343/… (the data has only been collected since Feb 2019). --- To see your own HNQs there's Rene's query at Data.SE which could be a starting point for gathering your own answer. – Rob May 7 at 18:42
  • 2
    @Rob Different question than that second link entirely. This is asking how sites are doing on the HNQ... that question is about a specific user. We don't track that, we do have ways of seeing this by looking at link clicks... and Glorfindel has a lot of data that we didn't track until recently, so we do have some historical info on which posts were on the HNQ. – Catija May 7 at 18:46
  • 1
    The closeness is that SE only recently started tracking, so asking for the data doesn't offer enough historical difference to draw a conclusion. Resorting to rewriting Rene's query or asking Glorfindel for historical data is helpful, but in the case of Glorfindel it's a 3rd party offering and not from SE (which is presumably the source origin requested). – Rob May 7 at 19:21
  • 1
    consider editing to ask for more hard data about these recent features - how many users turned off HNQ in their profiles, how many sites set custom question limits and title filters. With regards to custom limits, I haven't made my mind whether to count IPS or not: per my reading of their meta they rather firmly decided on 1-2 questions limit, but since they are still blocked from HNQ, does that disqualify them as data point – gnat May 8 at 8:48
  • @gnat IPS will definitely go back with 1 question soon(ish). I don't know if I should add anything about the number of questions, as there's already a separate question on that here. I'll add IPS there once we're back on HNQ. – Tinkeringbell May 8 at 9:19
  • @Tinkeringbell IPS is not of much interest to me, more like a tricky case of whether to count them or not. But I really would like to know if there were (other than IPS) sites that used available options to set custom titles filters and custom question limits – gnat May 8 at 10:22
  • @gnat IIRC there were a few other sites, such as Workplace and Christianity. Don't know, though. – Sonic the Anonymous WizHog May 11 at 10:05
  • @SonictheInclusiveHedgehog AFAIK none of these has custom limits on number of questions. You probably mixed it with obscure manipulations with hotness score on these sites (which by the way totally failed to do the job at Workplace) - it's discussed in more details here – gnat May 13 at 9:31
16
+100

Last week we completed the first 100 days since the HNQ changes went into effect. While we've made some other changes since then (e.g. minimum question age of eight hours), I figured that now is a good time to look at the numbers and see how this change has impacted the network. I'd like to look at them again when we hit six months or a year because, well... the HNQ is a fickle beast.

Right now, many sites look like they had huge drops or surges in their presence on the list but when you're sampling short time periods and the difference is one question either way because it's a low-volume site, it's going to look extreme. So, when you look at percentages, it's important to also look at the question count.

That said, we don't have a question count in SEDE from before this update due to not having a post history event that helped us track HNQs (for the pedantic, yes, that change went out a few days before but not long enough to be as useful here). So... we're kinda hampered in that but I'll get to this more later.

What I can look at in SEDE is the number of link clicks from the HNQ list and what percentage of total site directions those clicks are. The comparisons below are between the 100 days immediately before and after the changes to the HNQ list on March 11, 2019.

The top twenty sites by total HNQ traffic clicks before the update (fifth column):

                          % HNQ   % HNQ           CLICKS   CLICKS
SITE NAME                 BEFORE  AFTER  CHANGE   BEFORE   AFTER   CHANGE
The Workplace             17.7%   15.3%  -13.3%  1568566  1161275  -26.0%
Worldbuilding             19.2%   15.6%  -18.8%   657930   505964  -23.1%
Academia                  10.5%   10.1%   -4.1%   638862   529342  -17.1%
Sci-Fi & Fantasy           3.6%    4.8%   32.8%   475335   639892   34.6%
RPG                        3.3%    2.7%  -16.5%   440328   363327  -17.5%
Travel                     4.0%    4.1%    1.1%   395468   395603    0.0%
Info Security              3.7%    3.0%  -19.3%   365383   306157  -16.2%
Politics                  14.0%   15.3%    8.8%   313273   346610   10.6%
Money                      7.3%    8.8%   21.2%   303044   379524   25.2%
Stack Overflow             0.0%    0.0%   -6.7%   234866   224201   -4.5%
Physics                    1.7%    2.0%   16.2%   233108   243694    4.5%
English Language & Usage   0.5%    0.4%  -21.6%   232371   164796  -29.1%
Mathematics                0.4%    0.3%  -33.6%   229613   141442  -38.4%
Writing                   13.8%   14.2%    3.3%   223687   235209    5.2%
Skeptics                  11.3%    9.2%  -17.9%   206766   142507  -31.1%
Movies & TV                2.2%    4.3%   94.3%   201073   440532  119.1%
Puzzling                   8.5%    9.5%   11.6%   187167   187373    0.1%
Space Exploration         11.2%   11.0%   -1.7%   169263   158941   -6.1%
English Language Learners  1.0%    1.1%   12.6%   168525   216138   28.3%
Aviation                   3.9%    4.9%   26.4%   165470   251357   51.9%

Changes in traffic (clicks)

At the very top there are some big reductions, likely due to the cap of five questions at a time per site but as you move down things bounce around so that, within the top twenty, there's ten sites with fewer clicks, two about the same and eight that increase. Movies & TV, which more than doubled its views, and Sci-Fi largely saw a boost due to the release of several blockbuster films, particularly Avengers: Endgame (April 26, 2019).

On the other end, the site with the biggest reduction in clicks percentage and traffic is also a site that gets less than 1% of its traffic from the HNQ list - Mathematics. What's so interesting to me here is that Math has the highest number of questions on the HNQ list, so the clicks/question is quite low.

How has this impacted sites that get much of their traffic from the HNQ list?

I really love seeing which sites get the highest percentage of their clicks from the HNQ and, actually, the winner there both before and after the update is Retrocomputing, with 19.4% of traffic coming from the HNQ before the change and 23.7% after. They're not in the top 20 by total clicks as they have only ~100k before and ~150k after.

While Retrocomputing saw an increase, four of the ten sites that had >10% of their traffic from the HNQ list before the change saw a decrease of a few percentage points, particularly Computer Science Educators, which dropped from 15.4% to 10.5%, a loss of about 30% - but from the same quantity of questions - six. The total change in clicks is about 1200.

How many questions are we talking about?

Did these changes, for example, lead to differences in the number of questions on a site that hit HNQ

I mentioned before that I can't really get you SEDE data for this prior to the update. Fortunately, the marvelous Glorfindel had started tracking the HNQ list in August 2018, prior to these changes going into effect, so I do have some data, which he's shared with me. So, combining the two I get the following info:

More questions from the same number of sites

  • Before: 13,578 questions featured on 154 sites
  • After: 14,410 questions from 154 sites

This is an increase of 832 questions or 6.13%. This seems like a nice improvement! I'm guessing that a big part of this is maximum of three-days on the list we added.

How did question count change on a per-site basis?

Of the 170 eligible sites, 160 had at least one question on the HNQ list during the 200 days, which means 10 sites didn't appear on the list at all.

Of the 160 sites,

  • 76 sites saw an increase of two or more questions on the HNQ list,
  • 34 saw about the same amount of questions (change of -1 to 1)
  • 50 saw a reduction of two or more questions

The 160 sites individually had the following number of posts before and after the update - note that the buckets are a binary progression starting at 2 (because who doesn't like a little binary... plus the groups tend to squish together at the bottom):

                    Pre-HNQ   Post-HNQ
                    Change    Change
<2 questions          17        22
3-4 questions         13         8
5-8 questions         15        11
9-16 questions        17        18
17-32 questions       21        19
33-64 questions       19        20
65-128 questions      23        21
129-256 questions     14        20
257-512 questions     10        11
>513 questions         5         4

Which I made into a pretty chart:

Line chart showing the data in the table above.

For this time period, there's not a huge change in the quantity of sites with a certain number of questions that hit the HNQ. The line's a bit smoother but no drastic changes.

What does it all mean?

I've just showed you a bunch of numbers and they... don't really seem to show a huge overall change.

  • increase in the number of questions seen (YAY!)
  • more sites saw an increase in total questions than a reduction (YAY!)
  • same (pretty high) number of sites are getting featured (Meh)

There's a lot of other factors that we still can't really take into account to either truly quantify the effects but it's also difficult to qualify the impact (are the questions on the HNQ list better questions?).

  • numbers may be confounded by external factors/events we can't control
  • we don't know how long questions stay on the HNQ list (we don't measure it)
  • we don't know if the questions are generally better ambassadors for a site or more representative of the site quality.

Before I close, I want to talk a bit about the primary goals of the HNQ list update. The big ask was to help manage it a bit in a few key ways. Many of the features we implemented were designed to give control to moderators and users that we'd failed to implement and that was causing problems but also to add some tracking features. Additionally we set new limits on questions/site at a time and how long a question could stay on the list.

Moderator removal tool

While most sites haven't used this feature, some use it frequently. During these first 100 days 321 questions were removed from the HNQ list, mostly on the Mathematics site. Math seems to have the most questions hit the HNQ both before and after the update (1031 and 1010, respectively) but of the 1010 added during the first 100 days, 181 (17.9%) were removed by mods.

Site setting to remove the HNQ from the right sidebar

Is there a number of people who have used their profile settings to turn off HNQ entirely and is there any decline in traffic as a consequence?

Right now there's a bit over 1300 people using the "hide HNQ" setting. Not a huge amount in the grand scale of Stack Overflow but not bad for ~110 days. Comparing the total number of HNQ clicks before and after gives a gain of 211 clicks (10,201,659 before and 10,201,870 after). So... there doesn't seem to be a change in total clicks at all.

Conclusions

There's some ups and downs but, overall, I think the change has been a positive one both in the quantifiable changes and control of the HNQ list content. Even still, when reflecting on what the goals of the update were, I would call this a success.

  • leave the list events aren't really necessary for your purposes. Instead you can get reasonably approximated data by regularly taking snapshots of whole list, say, 3-5 times a day. We discussed this with Shog a while ago – gnat Jun 28 at 5:48
  • 1
    as for many questions from Math.SE, I think I got it. Thing is, they are enthusiastic about improving titles. Questions at sidebar catch attention of title editors and if their changes introduce MathJax this drops question from the list, so that next "candidate" takes its place - which catches the eye of editors again and next change follows, and so on. You can verify my hypothesis by checking how many of their hot question titles contain two or more dollar signs (ie MathJax). Given site topic I would expect there will be a lot – gnat Jun 28 at 6:53
  • have you (personally, or the team) got any changes, adjustments, clarifications regarding goals of HNQ? It doesn't look like these are firmly settled yet – gnat Jun 30 at 14:44
  • on a further thought, I think it would help to roughly estimate if the system picks sufficiently appropriate questions if you provide stats about how many hot questions get closed (and, for the sake of completeness, how many get closed then reopened). It is probably useful to have these stats per-site. Please consider adding some data on this matter - say, total network wide for the sake of completeness plus data on 20-30 top sites – gnat Jul 2 at 10:17
17
+100

Let's start off with this SEDE query showing the number of times the ♦ moderators have used the ability to remove a question from the HNQ list. It has been used about 120 times in the past 6-8 weeks; more than half of it by the Mathematics mods, but to put things in perspective (thanks @BillyMailman), they only remove about 10% of the questions which hit the HNQ list.

SEDE query results

  • That's a lot of questions removed from the HNQ. My gut is telling me that can't be right, but I don't know enough about Mathematics to say that for certain. Maybe someone, ideally a Mathematics mod, could shed some light? – Wrigglenite May 8 at 8:41
  • 1
    @Wrigglenite here is a list of affected questions. – Glorfindel May 8 at 8:49
  • 1
    And here should be the same list, but with the ones that weren't removed left in, to give a better idea what proportion are being removed (not as many as the 75 suggests; they get about ten Hot posts a day, on average) – Billy Mailman May 8 at 12:36
  • 1
    @BillyMailman good idea, I'll add a ratio column as well. – Glorfindel May 8 at 12:37
  • 4
    @BillyMailman it's worth keeping in mind that Math.SE folks have (and use) yet another way to drop questions from HNQ - by editing MathJax into their titles. This can be roughly estimated by counting how many of their hot questions contain two dollar signs in the title – gnat May 14 at 13:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .