9

Related:

I am looking for some statistics about 30-50 least exposed sites (those having least amount of hot questions) for recent 3-4 months.

Data I am primarily interested in is: average amount of hot questions per site per week, average amount of clicks per hot question and average number of views per these questions (I expect the latter to help estimate questions popularity besides HNQ sidebar).

For comparison purposes, I would also like to get the same stats for group of 80-100 sites having higher exposure and finally, for group of 20-40 most exposed sites.

I would also want to have these stats separately for IPS, in order to ensure that unique configuration of this site didn't skew the aggregate stats too much.

Note that proposed grouping (like 50-100-20) is not a strict requirement and if there is a different, more meaningful clustering, I will be OK with that.

Additionally, I would like to get stats about percentage of hot questions that were removed from the list by moderators, percentage of closed and / or deleted hot questions and percentage of protected hot questions (for sites grouped as described above). I doubt that these stats will be needed for the purposes of this analysis but want to make sure that we won't miss some useful correlation if there is any.

Improve this question
14
  • 2
    When I return from my holiday, I can have a look at the data my scraper has gathered. Any particular reason to look only at the last 3-4 months and not from the moment the HNQ saw major changes (which IIRC was in March this year)?
    – Glorfindel Mod
    Commented Nov 12, 2019 at 13:07
  • @Glorfindel thanks a lot! as for my selection of the timeframe, I added a solid delay from the recent changes to make sure that data is observed in "stabilized" environment, when site communities get used to the new way of working with recent features, that's why I preferred to exclude few first months
    – gnat
    Commented Nov 12, 2019 at 13:11
  • 2
    Is IPS really the only site to limit the number of HNQ questions at a time in such a way? I know IPS is kind of special, but I would have expected other site to say "we don't want more that 3 HNQ question at a time".
    – BelovedFool
    Commented Nov 12, 2019 at 13:22
  • @BelovedFool per my observations so far IPS is indeed the only site with custom limit (it's possible though that I could miss some)
    – gnat
    Commented Nov 12, 2019 at 13:24
  • 1
    I don't think Community Building has had any hot network question in the past 4 years.
    – gerrit
    Commented Nov 19, 2019 at 9:06
  • 1
    those having least amount of hot questions - what's the main criteria for you? Just the number of questions or should the duration in the HNQ matter as well? Math.SE is the site with the most # of questions in the last 4 months (904) but ranks only #19 on the 'number of spots, polled every 3 minutes' list. Worldbuilding.SE and RPG.SE dominate the latter one.
    – Glorfindel Mod
    Commented Dec 4, 2019 at 14:55
  • @Glorfindel just the number of questions ("least amount of hot questions"), because per my knowledge SE team folks abstain of estimating duration. Though now that you mentioned it, I recalled that your magical scripts can do estimates like that, so if you provide duration as additional details, that would be definitely great, it will provide additional dimension to learn about these things
    – gnat
    Commented Dec 4, 2019 at 15:26
  • My script records both entry and exit (well, the absence of a question in the list). Here is the # of hot questions from August 1st up to November 30th, and here the # of spots taken. Just let me know so that I'm comparing the correct site groups.
    – Glorfindel Mod
    Commented Dec 4, 2019 at 15:38
  • @Glorfindel Number of HNQs csv in your first link is what I am primarily after. If you could augment it with additional column containing spot percentage taken from second csv, that would be cool
    – gnat
    Commented Dec 4, 2019 at 15:49
  • 1
    Sure, will do. Expect an answer in 6-8 somethings :)
    – Glorfindel Mod
    Commented Dec 4, 2019 at 15:53
  • @Glorfindel after closer checking I noticed that tables have 150 entries - it looks like sites absent in there did not have HNQ at all, correct? FWIW my rough "personal clustering" in table based on number of questions is about like this (by rank, from top to bottom): rank 1 to 30, then 31 to 80, 81 to 108, 109 to 143, then the rest that didn't appear in HNQ at all (not much to learn about the latter, except for maybe their total amount)
    – gnat
    Commented Dec 5, 2019 at 13:03
  • 1
    Yes, that's correct. Including sites with no HNQs will make my queries a lot harder to write and decipher.
    – Glorfindel Mod
    Commented Dec 5, 2019 at 13:06
  • @Glorfindel would you have time to have a look on the data? I am considering starting a bounty on this - please let me know if you would prefer me to delay this a bit (recent integration experiment made me wonder if HNQ fail to properly educate users about other network sites and I decided it is maybe worth checking related data)
    – gnat
    Commented Apr 13, 2020 at 8:42
  • 1
    @gnat sure ... somehow I already wrote the query for this but didn't load the database with updated HNQ statistics. Will do so tomorrow...
    – Glorfindel Mod
    Commented Apr 13, 2020 at 20:08

2 Answers 2

Reset to default
8
+50

I have done some analysis about the period 2019-08-01 to 2020-03-31 which is summarized in this CSV. It is sorted by column 7, the number of views on Hot Network Questions per hour. The Workplace is the clear winner here with 364 views per hour, having almost double the number of views of the runner-up, Skeptics, with 188. Interpersonal Skills is number three in that list, so perhaps the unique configuration did help here, showcasing only the real gems on that site.

site                [-------BEFORE HNQ------]   [-------DURING HNQ------]   [Questions]     [Spots]
                    views   duration    views   views   duration    views   Ra  #   %       Ra  %
                                        /hour                       /hour   nk              nk
====================================================================================================
workplace           898     09:27:00    84.7    10157   36:20:00    364.9   17  488 1.65    8   3.04
skeptics            444     09:52:00    34.3    11023   59:51:00    188.4   74  85  0.29    38  0.89
interpersonal       398     10:07:00    33.9    8388    52:40:00    153.2   87  55  0.19    49  0.50
movies              138     09:00:00    14.3    4728    33:09:00    140.2   46  191 0.64    34  1.05
parenting           325     10:17:00    14.7    7057    52:26:00    139.4   101 40  0.13    65  0.36
money               269     09:44:00    21.1    5215    37:25:00    124.2   30  340 1.15    14  2.18
academia            277     09:17:00    23.9    4960    38:51:00    121.1   15  543 1.83    4   3.62
softwareengineering 183     09:20:00    19.4    4119    34:24:00    112.1   62  117 0.39    44  0.69
security            283     09:41:00    15.9    5691    46:50:00    109.0   43  215 0.72    21  1.72
politics            252     09:32:00    22.1    4387    36:37:00    105.6   18  487 1.64    7   3.04

Some clarification about the columns: the average Hot Network Question on Workplace needs 9 hours and 27 minutes to become an HNQ (slightly more than the minimum of 8 hours) and has 898 views in that period, which amounts to 84.7 views per hour. It spends (on average) 1.5 days in the HNQ and gets just over 10k views in that period. Not all of those views will be via the HNQ list, but if you subtract the views/hour from before the HNQ period, you'll get a decent estimate.

The last five columns are two ways of ranking all sites; by the number of distinct questions in the list, and the number of spots, i.e. how often the site occupies a place in the list. Of all Hot Network Questions in the network, 1.65% of those are Workplace questions, while if you pick a random Hot Network Question, it will be a Workplace question 3.04% of the time.

It's rather hard to say something about the least exposed sites; the list contains 156 entries, while there are about a dozen more sites which could appear in the HNQ but simply lacked the questions. Some of the sites who made the list had only one or two HNQs; not enough to perform a reliable analysis.

Data was gathered by downloading the Hot Network Question list every 3 minutes (similar to here). The main difference is that every 15 minutes, additional data about the questions is retrieved as well via the Stack Exchange API, including the number of views and the scores. Part of this data is loaded into a PostgreSQL database, of which a backup file is available here; all data relevant for this analysis is in the snapshots table.

As for your last paragraph, those events aren't registered in this setup, but you might be able to fetch some of the data from SEDE, since it records becoming an HNQ since March 2019.

Improve this answer
11
  • I can't see amount of questions in csv at github, there is only rank - do I miss something? (same for spots)
    – gnat
    Commented Apr 14, 2020 at 7:58
  • @gnat there's a corresponding percentage column. There were 29,664 HNQs in that period, and 11.2 million spots.
    – Glorfindel Mod
    Commented Apr 14, 2020 at 8:15
  • I saw percentage columns, but there is no column with total amount of questions so I can't calculate amounts per site, right? Or is the percentage calculated from total amount?
    – gnat
    Commented Apr 14, 2020 at 8:18
  • 1
    Yeah, you're right, I'm sorry. I can try to squeeze the # of questions in (# of spots is less relevant, I guess)
    – Glorfindel Mod
    Commented Apr 14, 2020 at 8:22
  • 1
    Yeah, you're right, I'm sorry. I've squeezed the # of questions in it now. (It can also be determined via SEDE, though it will miss questions which became hot shortly before 2019-08-01 but were still in the list at that date.)
    – Glorfindel Mod
    Commented Apr 14, 2020 at 8:38
  • the version with amount of questions is more convenient, thanks! (FWIW I managed to add it to previous version as a calculated column by multiplying percentages but it was harder to work with due to loss of precision)
    – gnat
    Commented Apr 14, 2020 at 9:08
  • 1
    You're welcome! The backup has been uploaded (with Git Large File Storage, never heard of it before) so it's possible to do your own analysis now.
    – Glorfindel Mod
    Commented Apr 14, 2020 at 9:09
  • I just noticed yet another thing, there seem to be no sites having zero questions in the csv, do I understand it correctly that every site in the network had at least one question in HNQ?
    – gnat
    Commented Apr 14, 2020 at 21:28
  • ...to re-check idea if there could be missing sites, I counted entries in csv and got 156 and compared it to amount of links at Stack Exchange sites list page which shows 174 links. It looks like 18 sites aren't in the list, could that be because these had no hot questions at all?
    – gnat
    Commented Apr 14, 2020 at 23:03
  • 1
    18 sites had no questions (some sites, like Meta, are excluded). It's rather hard to say something about the least exposed sites; the list contains 156 entries, while there are about a dozen more sites which could appear in the HNQ but simply lacked the questions. Some of the sites who made the list had only one or two HNQs;
    – Glorfindel Mod
    Commented Apr 15, 2020 at 5:40
  • I added overview of your data in a separate answer. If anything in there looks worth correcting, please let me know (or just edit it, I won't mind:)
    – gnat
    Commented Apr 28, 2020 at 12:58
3

Brief overview for those having no time to drill into details of data provided in top answer.

Data was collected for 8 months total between August 2019 and March 2020, inclusive.

There were 11 eligible sites that had no hot questions at all in this period: beer, coffee, conlang, ebooks, eosio, expressionengine, freelancing, languagelearning, tor, vegan, windowsphone. 30 more sites had less than one hot question a month average. 26 more sites had less than one hot question a week average.

On the opposite angle, 99 sites had one or more hot questions a week average. Of these, 46 sites had one or more hot question a day: chemistry, bicycles, gaming, superuser, or, boardgames, judaism, apple, retrocomputing, ell, dba, english, money, chess, space, codegolf, blender, askubuntu, japanese, gis, stats, salesforce, aviation, music, politics, workplace, diy, academia, codereview, mathoverflow, unix, travel, physics, stackoverflow, electronics, scifi, worldbuilding, mathematica, puzzling, rpg, tex, math.

In order to get some kind of aggregate data about less represented sites, I first reordered the data by amount of hot questions from smallest to largest, then calculated the rolling total. Next I scaled these absolute numbers to a range of 0 to 100% (since I wanted relative comparison).

Also, after observing huge variation in numbers I decided to cut off and ignore six top and six bottom sites assuming that these might be outliers. This took away quite a lot of original data (top six sites contributed about 25% of all hot questions, go figure), but remaining amount still looked large enough to build a meaningful picture.

Normalised total distribution on 156 sites looked as follows: 0%, 0, 0, 0, 0, 0.004506738, 0.009013475, 0.013520213, 0.01802695, 0.022533688, 0.031547163, 0.040560638, 0.049574113, 0.058587588, 0.067601064, 0.081121276, 0.094641489, 0.108161702, 0.126188652, 0.144215602, 0.162242553, 0.180269503, 0.202803191, 0.225336879, 0.247870566, 0.283924467, 0.319978368, 0.360539006, 0.401099644, 0.441660282, 0.48222092, 0.527288296, 0.576862409, 0.63094326, 0.685024111, 0.743611699, 0.802199288, 0.860786876, 0.919374465, 0.977962053, 1.041056379, 1.11316418, 1.203298932, 1.293433683, 1.383568435, 1.482716661, 1.599891838, 1.717067015, 1.83874893, 1.960430844, 2.100139709, 2.248862049, 2.402091126, 2.555320204, 2.713056019, 2.875298571, 3.042047862, 3.208797152, 3.384559917, 3.560322682, 3.740592185, 3.925368426, 4.128171617, 4.330974807, 4.542791473, 4.759114877, 4.984451755, 5.209788634, 5.43963225, 5.669475866, 5.90382622, 6.138176574, 6.38604714, 6.633917707, 6.881788273, 7.134165578, 7.391049619, 7.656947136, 7.945378341, 8.238316283, 8.535760963, 8.842219118, 9.157690748, 9.477669115, 9.806660958, 10.14466628, 10.48717833, 10.87025103, 11.25783046, 11.69047726, 12.14115102, 12.60534499, 13.07855244, 13.55626662, 14.0339808, 14.51620172, 15.03447654, 15.5572581, 16.0845464, 16.61183469, 17.14362973, 17.70697192, 18.30186128, 18.90125738, 19.50065348, 20.10455631, 20.75352652, 21.40700347, 22.10554779, 22.82211907, 23.57023751, 24.32736942, 25.0935148, 25.85966019, 26.68889991, 27.54968678, 28.47807472, 29.42448961, 30.39343819, 31.49308216, 32.59723286, 33.73293073, 34.8686286, 36.01784668, 37.17607824, 38.38839064, 39.63225021, 40.9617378, 42.32727928, 43.76042183, 45.25215197, 46.78444274, 48.33025373, 49.92113209, 51.67875975, 53.44540087, 55.25710938, 57.07332462, 58.98418135, 60.92207851, 62.96813737, 65.09982424, 67.2900987, 69.4848799, 71.68416783, 74.07724548, 76.52440398, 79.04817702, 81.72067241, 84.39767452, 87.11523728, 89.99954933, 93.08666456, 96.50727838, 100% (click here to see this data visualised to a chart)

From above, we can see that 86 least represented sites of 156 total contributed less than 10% hot questions, 106 contributed less than 20% questions and 135 - less than 50% questions.

21 top sites of 156 contributed over 50% hot questions. (Please keep in mind that this ignores top six and bottom six sites, otherwise the percentages for least exposed sites would look worse than here - as I mentioned above, ignored top 6 sites had about 25% of total questions.)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .