29

As some have noticed, the steward gold badge is now awarded repeatedly for every 1000 additional reviews in each queue.

Prologue

However, between the announcement and the change, the community focused its discussion entirely on the review queue redesign, while this specific change to the badge system - and its broader implications - went entirely unmentioned...

This post

Our community manager Catjia ♦ suggested in a comment that a post where the community might focus on this issue would be interesting:

You make a good point about parity with other badges for curation actions and I'd love to have a discussion of other badges that may benefit from being repeatable rather than one-time. I invite you to start a separate discussion or feature request here on Meta Stack Exchange to consider where else we can further recognize these valuable efforts by our community. No promises but I'd love to see the discussion.

I trust the community won't resist the invitation.

Status quo of badge gamification

As the state of the badge system has been stable for a long time, modifying the steward badge criteria is an impactful "game changer". The prior balance is significantly shifted by altering distribution and rarity of the gold badges while changing their proportion to silver and bronze badges.

Rarity of gold badges

Taking Stack Overflow as an example, this query gives some notable values:

Users per number of gold badges
Jon Skeet 783
250 82
500 62
750 52
1000 45
1500 37
2000 32
3000 26
4000 22
5000 19
6000 17
7000 16
8000 15
9000 14
10000 13

Shown in the following chart (top 200 users not featured):

chart of gold badges vs number of users

Parity of gold badges

One possible imparity:

  • Reviewers of a suggested edits would continue earning gold badges, while copy-editors arguably incurring in more work would be limited to earning one badge.

Some possible disparities:

  • Tag badges are certainly very hard to gain and only awarded once, while a steward badge can be gained linearly in 25 days - with comparably less work, depending on the queue.

  • Unlike before, it can become commonplace to see users having more gold badges than silver and bronze badges.

Role of the badge system

Below is a quote from the help center defining the two main roles of the badge system. Nevertheless one unmentioned practical role of the gold badges - as we know them - is indicating (together with reputation) at a glance of the user cards both the amount and kind of user participation. Removing the cap from the steward badges will render the user cards less informative, because they show some of their implicit information.

Quoting from the help center - What are badges?,

The badge system exists for two reasons:

  • to teach new users how Stack Exchange works
  • to encourage activities that are positive to the community

An additional definition may be added:

Stack Exchange badges fall into three broad categories:

Gold badges. Gold badges are for the most committed users. They reward the most difficult feats; you’ll have to not only participate in the community but be skilled, knowledgeable, and dedicated to earn these.

Time to earn gold badges

The number of daily reviews in any queue is limited to 40. This equates to the possibility of earning one steward badge every 25 days per queue, times 8 queues. Meaning, in the above SO example it would take an "ideally diligent reviewer" exactly 100 days of reviewing to reach the top-2000 users in number of gold badges (by comparison with former rules it took some of those users 6 to 8 years to reach their number of gold badges). In 200 days the hypothetical reviewer could reach the top-500, etc...

By comparison a single tag gold badge can be nearly impossible to gain in years.

Avenues for abusing the gold badge awards

Arguably, the defined role of the gold badges "They reward the most difficult feats" could loose its meaning; The simplest example would be doing FP reviews only upvoting comments.

Epilogue

This post is not meant to be exhaustive. Quite the contrary, its intention is focusing any discussion and ideas the community may want to contribute on the matter.

7
  • 4
    Should additional gold badges be awarded for twice as many reviews as the prior one, example: 1K, 2K, 4K reviews awards three gold badges for 7000 reviews or (for example) every additional 2500 reviews - or would you suggest elimination or a cap? --- Is there a proposed solution from the question asker, or simply the statement that: "this is a problem"?
    – Rob
    Jan 29 at 17:00
  • @Rob some people have mentioned that in chat, that perhaps award levels should progressively increase.
    – bad_coder
    Jan 29 at 17:03
  • Could we have a link, I don't see it in The Tavern. --- One problem I see, not mentioned, with making the additional badges available is that it takes reviews (and badges) away from others.
    – Rob
    Jan 29 at 17:05
  • @Rob I prefer not linking to individual chat messages. That "one problem" you mention is interesting and I hadn't thought about it, consider adding it as an answer.
    – bad_coder
    Jan 29 at 17:30
  • 12
    I will now be the recipient of dozens of gold steward badges. They will be more than half of my gold badge total. I don't really need that, it seems silly, & I do think it dilutes the signal from the badges / user card. The idea that each additional badge should require 2x the prior number of reviews seems like a good compromise. Jan 29 at 18:22
  • The daily limit of reviews on math.se is 20 reviews per day. So I'm not sure what site you are referring to. On math.se, it takes 50 days, reviewing 20 posts, to earn a gold medal, and I'll note that users seldom exhaust their allotted downvotes for reopen queue, and late answers, particularly because math.se has a lot of reviewers.
    – amWhy
    Feb 16 at 23:02
24

That of course assumes that the value of a gold badge is 'fixed', and what you consider the badge system. I tend to consider it almost a "new game plus" or Achievement system.

I have 28 gold badges on meta at the moment

enter image description here

I don’t particularly think any of these, except maybe electorate, took any real extra effort on my part. Each of these commemorates a *single* post of distinction rather than applied effort. Even legendary, to me involves a certain degree of luck.

Steward needs actual work.

1000 reviews done right is a *lot*

The number of daily reviews in any queue is limited to 40. This equates to the possibility of earning one steward badge every 25 days per queue, times 8 queues

Which indicates a non trivial expenditure of time, unlimited reviews …. in short, you’re presupposing an ideal spherical cow, when actual cows come in many sizes. That’s even assuming they have enough posts to review, which many smaller sites might not.

If someone is willing to put in a non trivial amount of time in the review queues, helping make things better, well, maybe they deserve that gold badge.

2
  • 4
    This post is not exactly true, writing 200 non-wiki answers technical answers isn't a lot of work (compared to reviewing)?
    – bad_coder
    Jan 29 at 16:52
  • 2
    But you're going to do it anyway, and if you need extrinsic motivation - there's reputation
    – Journeyman Geek Mod
    Jan 30 at 1:31
15

As a PM for the Stack Overflow Public Platform team, I want to build upon Journeyman Geek's points. One of the goals of the Review Queues overhaul is to encourage and reward reviewing behaviors. We realized that in awarding the Steward badge only one time, we weren't providing much of an incentive to the users who dedicate significant time and energy to review tasks.

In terms of impact, on Stack Overflow, only 11% of current Steward badge holders will go on now to receive multiple Steward badges. Across the rest of the Stack Exchange network, excluding Stack Overflow, the average is 23%.

When it comes to total number of Steward badges awarded, it increased by 35% on Stack Overflow and by 83% on average across the network.

5
  • 3
    Well seeing that progress bar completely filled, and then some more, was definetly not encouraging me to do any more reviews. So great to see more badges will come. Perhaos there are other similar actions that need to be rewarded in the same fashion.
    – Luuklag
    Jan 29 at 18:49
  • 2
    The total number of Steward badges has already doubled on my site, & we haven't reached the end. Jan 29 at 21:21
  • 1
    Maybe the goal should be made clear: when awarding the last badge on 1000 reviews, I would assume that SO's "brain" was expecting to have some rotation. That is, that instead of having the same people reviewing over and over, new ones would replace and provide their new point of view. The fact that this changes makes me think that the "brain" sees that there are not enough new reviewers, so it is time to re-engage the old ones. Jan 30 at 13:24
  • Hello @AnitaTaylor, this data is invaluable and helps the community paint a clear picture. Thank you for listening and participating. Please understand that writing the post was extraneous, I've researched the matter further (outside SE and the thousands of threads within). Understandably folks at product management might've preferred a different timing, while the community only just noticed this and may have their own rhythm in reacting and contributing their analyses.
    – bad_coder
    Feb 1 at 4:05
  • 3
    @fedorqui'SOstopharming' Our goal is also to encourage more new reviewers to participate. One of the next phases of Review Queue improvements is to improve onboarding, so that newer participants are aware of Review Queues and how to participate in each queue
    – Anita Taylor StaffMod
    Feb 1 at 15:24
9

Starting with an addendum to

However, due to specifics of Stack Overflow's huge Close Vote Queue and the fact that close votes age, I am mostly close voting outside review queue.

That is true for me, too.

But the "surprising" other steward badge that came in last week ... that notification simply brought me back to look at the review queues (on stackoverflow) since ages.

Long story short, I consider this a very positive move, and together with the other "overhauling" of review UIs and processes, I think it will have quite a revitalizing effect how I spent my time on stackoverflow, and it will most likely lead to me spending more time there doing curation work (the other part that drives my motivation there is the sad fact that curation becomes more and more important, as the number of low quality input is ever increasing it seems).

And after using some of the updated review UIs, yes, this is definitely going in the right direction.

5
  • 1
    LOL, your signature in this post: "Sincerely, GhostCat 8th Steward of H&I" if memory serves me right I was the 84th H&I steward, funny title :D
    – bad_coder
    Feb 1 at 8:11
  • 2
    @bad_coder Fun fact: my counter for the H&I queue still says "1000". That is the only queue were I decided to not advance that counter, even by 1.
    – GhostCat
    Feb 1 at 8:15
  • I also have exactly 1000 in H&I, there aren't enough posts in the queue for everyone.
    – bad_coder
    Feb 1 at 8:35
  • 3
    It can be a lot easier to vote to close in subjects you're familiar with, too, which is why we're doing a lot to help y'all sort those 3-5k posts that need review down to ones that you have some expertise in - I know that there's some SOBotics/SOCVR tools that will help identify these posts - but we'd love to do more in-house and in the UI so that y'all can really focus and get reviews you feel able to address.
    – Catija StaffMod
    Feb 2 at 14:42
  • Sounds good. Whatever makes the job easier and less frustrating, the better.
    – GhostCat
    Feb 2 at 14:47
4


Since most of the answers (and even the question to some extant) seem slanted towards high traffic sites/queues, I thought I might give a perspective from someone who is active on sites that do not get anywhere near the numbers in their review queues as the busier(st) StackExchange sites.

I personally believe this is a nice incentive for those few 'super reviewers' who like to jump in and really get things done. For most though, even the first 1000 barrier is pretty much far out of sight. It would take years to get one on even the busiest queue of these lower traffic sites. So these additional badges are meaningless and won't affect their participation one bit. They'll still drop into the queues every now and then for whatever personal motivation and help out, but that's it.

So that leads to my feature request. Is there any plausible way that Beta sites could setup their own milestones for participation in the queues? This might just actually get users to participate usefully and more often, and give them the right kind of positive feedback for their efforts (which on the smaller sites - can involve a lot of patience and perseverance).

7
  • Arent these called bronze and silver badges...
    – Luuklag
    Feb 5 at 12:20
  • 2
    @Luuklag, On the Beta sites, there are already lower thresholds for obtaining privileges. So obviously the thresholds for all badges would be adjusted accordingly. Also, a gold badge is obviously more reflective (meant to be) of the work put into getting it than a silver badge, which is likewise more reflective than a bronze badge. This is stated in the description of the badges themselves. On the lower-trafficked sites getting a silver badge in the busiest review queue could literally take years of dedication. Since the OP brought up 'parity', then it seems fair to point out that there is...
    – ouflak
    Feb 5 at 12:26
  • ... a lack of such here with regards to the smaller sites, which will effectively not have the opportunity to benefit from the good intentions of this change, or this kind of change. If that is also desired for the smaller sites, some adjustment has to be made.
    – ouflak
    Feb 5 at 12:27
  • If you are going to change the amount of work required to earn a certain gold badge, as that is what your proposal boils down to, you are diminishing its value in comparison to the rest of the network. I'm very opposed to that. It takes the same amount of work to 1000 reviews on SO as it does on any other site.
    – Luuklag
    Feb 5 at 12:54
  • 1
    @Luuklah, Actually, that's kind of my point. The amount of work is relative to the site upon which the work is done. And keep in mind, that the 'amount of work' is already different for privileges. Some of those privileges even include access to the review queues. So StackExchange has already acknowledge this disparity. I'm merely requesting that we extend this acknowledgement to the review badges as well to be consistent.
    – ouflak
    Feb 5 at 13:14
  • 1
    And I'm sure you didn't mean to imply otherwise, but there is a lot of work that goes into maintaining a smaller site. I partake in the upkeep of both high-traffic and low-traffic sites, so I know that there is a difference. The bigger sites naturally have a LOT more overhead. But it's still work. There's nothing wrong with being consistent in rewarding contributors for it.
    – ouflak
    Feb 5 at 13:15
  • 2
    @ouflak This proposal is innovative, it raises the issue of inter-site badge parity at a time homogeneous intra-badge parity was being discussed. Which is a rare and opportune POV on the issue, I haven't seen this mentioned before (the thought will come as a surprise to many.) Good post!!
    – bad_coder
    Feb 5 at 16:23
3

Awarding multiple gold badges for the same queue is nice :)

However, due to specifics of Stack Overflow's huge Close Vote Queue and the fact that close votes age, I am mostly close voting outside review queue. When I was starting most of my close votes cast in the review queue didn't make a difference and eventually questions were not rightfully closed.

At the moment I have more than 32.000 total close votes cast and only about 5.600 was done through CV review queue.

My proposal is to award Steward badges based on number of total close votes cast and not only for those casted through review. Same principle could be applied on Reopen queue.

6
  • 5
    I'm generally not a fan for having any kind of reward just for voting (be that badges or hats). Too often, this ties the voting to other things than post quality. For close votes, right now the count I see on my profile includes questions that never ended up closed, and votes that aged away. If there are going to be badges, please only award them when the question actually ends up closed or reopened.
    – Tinkeringbell Mod
    Jan 30 at 16:12
  • 1
    @Tinkeringbell I don't mind awarding only "good votes". I just don't think that differentiating actions between queues and outside queues is a good thing. Jan 30 at 18:05
  • 1
    Well I have to disagree here. For a user with the close vote privilege it might indeed be easier to vote outside of the review queue. I do it all the time. But take a step back and consider the larger picture. What did you do before you were able to close vote? You flagged posts, right! So what happens to those flags? They end up in the review queue. I think we should really be reviewing more, so that new(er) users that flag posts see that they can actually make a difference by doing so. This will give them faith in the system and hopefully make them effective close voters when they can.
    – Luuklag
    Feb 1 at 8:08
  • 1
    @Luuklag I would not mind going through review queue if I would see some results. When I was doing so, most of my close votes were useless because question was never closed. And I am not talking about border line questions, but totally off topic ones. I don't really care about badges, but if badges are being awarded then it does not seem fair that they only count if you go through queue. Anyway, I am not going to use CV queue again, just because I would get a badge. Feb 1 at 8:52
  • At least now only 3 CV's are needed to close a Q, so chaces of your votes expiring should get lower.
    – Luuklag
    Feb 1 at 9:17
  • @Luuklag Yes, I tested it... still not good enough. It was better for the moment while Samuel Liew was daily going through the queue and keeping it in check, but now with 4+K questions CV-s mostly just get lost. Feb 1 at 9:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .