Gold badge parity and balance - award criteria of the steward badge changes

Question

As some have noticed, the steward gold badge is now awarded repeatedly for every 1000 additional reviews in each queue.

Prologue

• The first mention of (possible) upcoming changes to the badges came with the "Dark Mode announcement" - 30/3/2020.
• Next a proposal was made "Improving Review Queues - Design overview II: Changing review bans and other new features" - 23/4/2020.
• Finally, today the change becoming effective was announced "Visual design changes to the review queues" - 28/1/2020

However, between the announcement and the change, the community focused its discussion entirely on the review queue redesign, while this specific change to the badge system - and its broader implications - went entirely unmentioned...

This post

Our community manager Catjia ♦ suggested in a comment that a post where the community might focus on this issue would be interesting:

You make a good point about parity with other badges for curation actions and I'd love to have a discussion of other badges that may benefit from being repeatable rather than one-time. I invite you to start a separate discussion or feature request here on Meta Stack Exchange to consider where else we can further recognize these valuable efforts by our community. No promises but I'd love to see the discussion.

I trust the community won't resist the invitation.

Status quo of badge gamification

As the state of the badge system has been stable for a long time, modifying the steward badge criteria is an impactful "game changer". The prior balance is significantly shifted by altering distribution and rarity of the gold badges while changing their proportion to silver and bronze badges.

Rarity of gold badges

Taking Stack Overflow as an example, this query gives some notable values:

Users per	number of gold badges
Jon Skeet	783
250	82
500	62
750	52
1000	45
1500	37
2000	32
3000	26
4000	22
5000	19
6000	17
7000	16
8000	15
9000	14
10000	13

Shown in the following chart (top 200 users not featured):

Parity of gold badges

One possible imparity:

• Reviewers of a suggested edits would continue earning gold badges, while copy-editors arguably incurring in more work would be limited to earning one badge.

Some possible disparities:

• Tag badges are certainly very hard to gain and only awarded once, while a steward badge can be gained linearly in 25 days - with comparably less work, depending on the queue.

• Unlike before, it can become commonplace to see users having more gold badges than silver and bronze badges.

Role of the badge system

Below is a quote from the help center defining the two main roles of the badge system. Nevertheless one unmentioned practical role of the gold badges - as we know them - is indicating (together with reputation) at a glance of the user cards both the amount and kind of user participation. Removing the cap from the steward badges will render the user cards less informative, because they show some of their implicit information.

Quoting from the help center - What are badges?,

The badge system exists for two reasons:

• to teach new users how Stack Exchange works
• to encourage activities that are positive to the community

An additional definition may be added:

Stack Exchange badges fall into three broad categories:

Gold badges. Gold badges are for the most committed users. They reward the most difficult feats; you’ll have to not only participate in the community but be skilled, knowledgeable, and dedicated to earn these.

Time to earn gold badges

The number of daily reviews in any queue is limited to 40. This equates to the possibility of earning one steward badge every 25 days per queue, times 8 queues. Meaning, in the above SO example it would take an "ideally diligent reviewer" exactly 100 days of reviewing to reach the top-2000 users in number of gold badges (by comparison with former rules it took some of those users 6 to 8 years to reach their number of gold badges). In 200 days the hypothetical reviewer could reach the top-500, etc...

By comparison a single tag gold badge can be nearly impossible to gain in years.

Avenues for abusing the gold badge awards

Arguably, the defined role of the gold badges "They reward the most difficult feats" could loose its meaning; The simplest example would be doing FP reviews only upvoting comments.

Epilogue

This post is not meant to be exhaustive. Quite the contrary, its intention is focusing any discussion and ideas the community may want to contribute on the matter.

Answer 1

That of course assumes that the value of a gold badge is 'fixed', and what you consider the badge system. I tend to consider it almost a "new game plus" or Achievement system.

I have 28 gold badges on meta at the moment

I don’t particularly think any of these, except maybe electorate, took any real extra effort on my part. Each of these commemorates a *single* post of distinction rather than applied effort. Even legendary, to me involves a certain degree of luck.

Steward needs actual work.

1000 reviews done right is a *lot*

The number of daily reviews in any queue is limited to 40. This equates to the possibility of earning one steward badge every 25 days per queue, times 8 queues

Which indicates a non trivial expenditure of time, unlimited reviews …. in short, you’re presupposing an ideal spherical cow, when actual cows come in many sizes. That’s even assuming they have enough posts to review, which many smaller sites might not.

If someone is willing to put in a non trivial amount of time in the review queues, helping make things better, well, maybe they deserve that gold badge.

Answer 2

As a PM for the Stack Overflow Public Platform team, I want to build upon Journeyman Geek's points. One of the goals of the Review Queues overhaul is to encourage and reward reviewing behaviors. We realized that in awarding the Steward badge only one time, we weren't providing much of an incentive to the users who dedicate significant time and energy to review tasks.

In terms of impact, on Stack Overflow, only 11% of current Steward badge holders will go on now to receive multiple Steward badges. Across the rest of the Stack Exchange network, excluding Stack Overflow, the average is 23%.

When it comes to total number of Steward badges awarded, it increased by 35% on Stack Overflow and by 83% on average across the network.

Answer 3

Starting with an addendum to

However, due to specifics of Stack Overflow's huge Close Vote Queue and the fact that close votes age, I am mostly close voting outside review queue.

That is true for me, too.

But the "surprising" other steward badge that came in last week ... that notification simply brought me back to look at the review queues (on stackoverflow) since ages.

Long story short, I consider this a very positive move, and together with the other "overhauling" of review UIs and processes, I think it will have quite a revitalizing effect how I spent my time on stackoverflow, and it will most likely lead to me spending more time there doing curation work (the other part that drives my motivation there is the sad fact that curation becomes more and more important, as the number of low quality input is ever increasing it seems).

And after using some of the updated review UIs, yes, this is definitely going in the right direction.

Answer 4

feature-request
Since most of the answers (and even the question to some extant) seem slanted towards high traffic sites/queues, I thought I might give a perspective from someone who is active on sites that do not get anywhere near the numbers in their review queues as the busier(st) StackExchange sites.

I personally believe this is a nice incentive for those few 'super reviewers' who like to jump in and really get things done. For most though, even the first 1000 barrier is pretty much far out of sight. It would take years to get one on even the busiest queue of these lower traffic sites. So these additional badges are meaningless and won't affect their participation one bit. They'll still drop into the queues every now and then for whatever personal motivation and help out, but that's it.

So that leads to my feature request. Is there any plausible way that Beta sites could setup their own milestones for participation in the queues? This might just actually get users to participate usefully and more often, and give them the right kind of positive feedback for their efforts (which on the smaller sites - can involve a lot of patience and perseverance).

Answer 5

feature-request

Awarding multiple gold badges for the same queue is nice :)

However, due to specifics of Stack Overflow's huge Close Vote Queue and the fact that close votes age, I am mostly close voting outside review queue. When I was starting most of my close votes cast in the review queue didn't make a difference and eventually questions were not rightfully closed.

At the moment I have more than 32.000 total close votes cast and only about 5.600 was done through CV review queue.

My proposal is to award Steward badges based on number of total close votes cast and not only for those casted through review. Same principle could be applied on Reopen queue.