"Generalist" is a badge awarded to community members for broad knowledge of a site's most commonly-asked-about subjects. It's earned by providing answers that accumulate 15 total score per tag in 20 of the top 40 tags. But, there's a caveat - in order for the Generalist badge to be awarded, there must be 40 "top tags" - defined, in this case, as a tag used on at least 200 questions.
It turns out, this is a major inhibitor for many sites (not just small ones!) to even start awarding this badge, as I found out when investigating a question on Retrocomputing Meta: Why has nobody ever been awarded the Generalist badge? If you do the same back-of-napkin math I did, you discover that for the badge to even be awarded, there must be at least 1600 questions (and that assumes that those same 40 tags are used equally across those questions, five tags each i.e. 40/5 x 200). Most of the time, these tags are used only one or two per question, making the necessary number of questions much higher.
For sites at the scale of Stack Overflow or even some of the large to mid-sized Stack Exchange sites, that's completely doable, but for smaller sites that are sitting between 2000-8000 questions, it's nearly impossible. One of the reasons that we set these limits is that we wanted to ensure that the "top tags" on a site had stabilized somewhat before starting to hand out the badge - and that's a reasonable intention - but we wondered whether 200 questions each in 40 tags was the right number to determine that.
As such, I explained all of this to Slate and asked her to use her magic SQL skills to dig into the data on SEDE. Essentially, my question was this: what would be the impact of adjusting the number of questions/tag needed (currently 200) and/or the number of total tags with that many questions (currently 40)? I dropped this all in her lap before going on vacation for three weeks - thanks, Slate!
When I got back, she'd prepared a very nice suite of data for me. She'd looked at several cases, each one step away from the other:
- The status quo - 40 tags with 200 questions each.
- Require fewer "top tags" - 20 tags with 200 questions each. (Note: we can't go below 20, because we're not re-evaluating the "score of 15 in each of 20 top tags" criterion.)
- Require fewer questions per "top tag" - 20 tags with 100 questions each
- Require even fewer questions per "top tag" - 20 tags with 50 questions each
In each case, we didn't change the requirement for the badge itself - it was still necessary for a user to get a score of 15 in 20 of the top tags ("top tags is defined as the 40 most-used tags with at least n questions in that tag).
She re-ran the data using these new triggers to see the impact of how many users per site would have earned the Generalist badge and the data was pretty awesome. For the larger sites, nothing changed - if site already met the qualifications for the higher requirements, no new badges are awarded, but for sites that don't meet the requirements, there was a sizable increase.
With the current badge requirements, only about 75 sites had even one user getting this badge (brown line below). With the others, as many as 150 sites had at least one Generalist - leaving only a couple dozen sites on the network unable to award the badge.
An important note about the graph above - while it looks like there's a change in the number of badges awarded for the first 25 sites, this is deceptive - none of the sites with any Generalist badge winners saw any additional qualifying users. When we order the list by volume of awardees on a per-case basis. In some cases, sites that had no badges awarded end up with more badge recipients than sites that currently have them, so the "Site Number" in the graph above doesn't always refer to the same Stack Exchange site, just the count of badges on the nth ranked site.
Because we knew that there wouldn't be a change in awards on the bigger sites and to better show the value of these changes on smaller sites, the graph below excludes all sites with zero change in awardees - so all of the sites that currently have any as well as the sites that have zero, both before and after the change. This graph shows, of sites that see a change in awardees, how many awards occur on those sites.
In the most limited change, about thirty new sites would see at least one person awarded the badge. Reducing the number of questions to 100 increases that to just over fifty sites and further reducing the questions needed per tag to 50 increases the count to nearly eighty sites.
To make it easier to see, the graph above was cropped at 50. One site, Code Golf & Coding Challenges, will end up with nearly 150 members earning this badge if these changes are made.
Based on the data, we're proposing to change the requirements for awarding the Generalist badge to the following:
- Provide answers that accumulate 15 total score per tag in 20 of the top 40 tags (unchanged)
- We lower the number of questions required for a tag to count as a "top tag" to 50.
- We remove the requirement for there to be 40 "top tags" with at least 50 questions each. Because the badge requires there to be at least 20 tags with 50 questions, it's similar to lowering the requirement to 20 but a simpler query.
- In other words, for example, if the site only has 30 tags being used on at least 50 questions, and a user earns a score of 15 on 20 out of those 30 tags, they would now qualify for the badge. If the site only has 20 tags being used on at least 50 questions, a user would qualify if they have a score of 15 on all 20 of those tags.
As a note, only tags with at least 50 questions will be considered "top tags" and only the top 40 tags will qualify to earn the badge, even if there are more tags with at least 50 questions each.
Given the data above, do you have any concerns about this change? Are there other variations of this that we should consider before changing this?
For those of you who speak SQL, I've shown the query below for the before and after for the proposed change. While I can't guarantee this is the specific way the devs will change the badge query, the end result should be the same and we'll update y'all with what the devs end up doing if we do make this change live.
On the left - the SQL (roughly) as it is now. On the right - what we're proposing it'll become. The parts that differ are highlighted in bold.