How should a newly graduated site decide when to request reputation threshold changes?

Question

SE site graduation used to involve a bunch of things happening at the same time: removal of the "beta" label, a shiny new design, privilege thresholds increasing from beta levels, inaugural moderator elections, and a few other bits and pieces. Since design-independent graduation kicked off in 2015, these different facets of graduation have become increasingly disconnected from each other, and when graduated sites get their designs and rep threshold changes can vary widely.

In the last couple of years, a vast number of SE sites have quite suddenly "graduated" (i.e. had their beta label removed, and in some cases no other changes), leaving very few beta sites remaining:

• Congratulations to our 29 oldest beta sites - They're now no longer beta!
• Congratulations to the 59 sites that just left Beta

Many of these sites, and earlier graduated ones too, are still lacking other features of graduation. Obviously, getting a design depends on SE designer time, and getting mod elections depends on a bunch of factors including CM time, but changing reputation thresholds is presumably (from SE's point of view) just a matter of a simple button push, which could be requested by the community on meta and implemented with less delay.

The purpose of this post is to figure out guidelines for when a site/community should request this. Some pros and cons are listed in the PPCG meta that I linked earlier. This is an important part of graduation and could help a site to feel "mature", much like the beta label removal, and would (at least in theory) restrict participation in moderation activities like closing/deleting to those with more experience and knowledge of the site. On the other hand, it would drastically reduce the number of people able to perform those activities, which for some smaller graduated sites could be crippling.

What metrics or statistics would be useful to check? Number of users (or of active users, by some definition of "active") currently above certain rep thresholds? Number of reviews and active reviewers per year? Percentage of questions getting closed and answers getting deleted? Anything else?

Answer 1

This is something that the CM team has been thinking about since before I pushed that first set of 29 sites out of beta in 2019. It was, in fact, part of our conversation when we made the decision to do that. One of the things that we've noticed and reiterated nearly every time we've made changes to the Graduation process is that we recognize that there's not one "right" site size and that not every site will be the size of Stack Overflow.

That's no less true today than it was in 2015 when Ana wrote Graduation, site closure, and a clearer outlook on the health of SE sites or I announced those 29 sites leaving beta in 2019 or Rosie announced the round of 59 sites at the end of 2021.

You recognized the same thing that we did, though - we have more and more sites out of beta and some of those sites are even getting site designs… and we asked ourselves… "Where do the 'graduated' or 'full' reputation privileges come in now?" It's not happening when they leave beta… it's not happening when they get a design… so when?

We asked ourselves the same questions you're asking - what metrics would we outline for when we would reach out to a site and say, "Congrats, it's time to have higher reputation levels!" or when a site could come to us and ask "We can haz higher rep levels please?" And the answer that we came to is - if it's not broken, let's not "fix" it. That is to say…

We have no plans to change reputation levels across the board on a site from "beta" to "full" ever. We will only adjust individual reputation levels if the current number is not working.

Now, whether this decision excites you or makes you furious, let's take a minute and work through why we've made this decision.

Site age != big community != lots of rep

When sites moved through the process more quickly, there was often more content being created, more votes being cast and more people with the reputation needed to get the higher reputation levels. But as sites have focused on smaller and more niche subjects with fewer experts to participate and judge content, we realized that those "full" rep levels might never be a good fit, just as getting to 10 questions per day might never be possible. But - that doesn't mean the community isn't sound, healthy, and serving a valid purpose.

Since the last time we gave people higher rep levels, I can only think of one site that's asked for them, maybe two… and I can guess why that is.

Losing privileges is a bummer.

Losing privileges is frustrating to people. I see people complaining about it often - paying for bounties causes them to lose editing privileges or posting an unpopular answer drops them below the reputation needed to see deleted posts or downvoting loses them the privilege to downvote (probably my favorite)... It's just not fun for anyone involved. Heck, it's why we don't take away badges (in most cases) even if you would lose them otherwise.

A bunch of people losing a whole raft of privileges because a subset of users decided it was time to require higher privileges on a site… just doesn't make for great community happiness and can, actually, lead to poorer community engagement. Particularly in cases where there really weren't enough people with the privileges to do tasks that required several participants.

Honestly, I feel like a big part of why we tested 3-vote closure (and it's been so popular on sites) is because many of those big sites with higher rep levels don't have enough people with the close privilege who are interested in participating in that task. Imagine doing that to smaller sites that only have 5-10 active people with 3k reputation. The people who lose the privilege aren't necessarily going to be motivated to earn it back, and may just wander off.

Why are we still using reputation to determine access to privileges anyway?

If you read much of what I post on meta, you may have heard me opine about how we need to stop locking privileges to reputation, anyway. Someone with 100k reputation doesn't necessarily know how to edit a post well but someone with 500 reputation who's had their last 100 suggestions accepted probably does. In a perfect world, we would much rather give access to tools to people who have shown through some method that they know what they're doing - and make sure they continue to perform well at those tasks, lest they lose that privilege.

While we don't have it on our roadmap to start this work, right now we don't think it makes sense to further push the rhetoric of reputation being equivalent to ability on these sites by making people re-earn privileges they've already shown they're more than capable of using well.

Our site has an influx of crappy tags from low-rep users… we need higher privilege levels!

Whether it's tags or close votes or post deletion, this is where the second half of the big bold header up top comes in. "We will only adjust individual reputation levels if the current number is not working." So, the narrative here isn't "you're just going to have to deal"; it's - "OK, tell us about the problem you're facing". If your site has newer users going on tag creation sprees or voting to leave open questions that should be closed, that's bad! Let's see what we can do about it - and this includes cases where the "full" privilege levels may be too high.

Whatever the privilege, we can adjust it on a per-site basis without changing the entire site to the full site privilege levels. This is why Stack Overflow has even higher levels for some things - including tag creation. We want to target the specific problem your community is facing rather than having potentially negative impacts on all privileges because of an issue with one or two.

While I can't say we'll never change sites from beta to full rep levels in the future, sites should be aware that it's unlikely we will. If closing and reopening are working fine at 500, what's the purpose of flipping it to 3k? If there is an issue, we can address it; otherwise, let people do their thing!

For a full listing of the privileges and exceptions, check out What are the reputation requirements for privileges on sites, and how do they differ per site?

Answer 2

What metrics or statistics would be useful to check? Number of users (or of active users, by some definition of "active") currently above certain rep thresholds? Number of reviews and active reviewers per year? Percentage of questions getting closed and answers getting deleted? Anything else?

What you're describing seems to me to be a very global look at the site.

I would suggest looking at each privilege individually, consider who uses that privilege in the current situation and then see how they would be affected by the new situation.

Let's take close vote reviews as a practical example. I would look at all the close vote reviewers over the past year. Then make a list (or bar chart) and sort them in descending order of how many close vote reviews they did over that period. So at the start we get the most active reviewers while the end consists of a tail of users who completed only a few reviews that year.

In my head it looks like this power law graph with the most frequent reviewers on the left and the least frequent ones on the right. I don't know if it actually looks like that, but online participation (like many other data sets in social sciences) tend to follow such a distribution.

Let's say we color all the bars blue indicating the corresponding users are able to review at the current rep levels. If we increase the rep requirement, some of those users won't be able to review. In my imagination we'd have a slider in which we can virtually increase the reputation requirement and those users who fall off would turn red. With that, you should be able to see if a proposed increase leaves enough recently active reviewers to keep the site running. How many active reviewers may lose their privilege for the change to be successful is up for debate, but I think such a visualization is a good option to get the practical implication in one graph.

Another factor in play is how many active reviewers are actually available. If there are only 6 active reviewers then losing 2 might be a big burden on the site because it hampers the ability to close questions. If you had 60 active reviewers then losing 20 (perhaps only temporarily) wouldn't be as bad.

One downside of this approach is that it only considers the current user base. With beta rep requirements, new users can get into reviewing after a reasonable bit of participation. With graduated reputation requirements it will take much longer for new users to be allowed to review. That's an aspect to consider, but the bar chart approach doesn't account for that.

Another thing to consider is that some of those users who lose their privilege due to the reputation requirement change may be close to regaining the privilege. So you could make another progress bar chart to show for those who would be cut off, how far they are toward regaining the privilege. This could all be incorporated in the same graph by using some color gradient for those who lose the privilege.

Answer 3

For a site to be in good health we really want a significant part of the moderation such as reviewing and editing to be handled by non-moderators. If we don't have that then we really don't have community participation, we just have moderator participation and they close and reopen by fiat.

So we'd need to see what effect changing the close/reopen privilege would have. How many of the existing close/reopen voters would we lose? Are those that would remain, active reviewers? Would we end up having the same very few people close and reopen everything?

Similarly with editing, would we turn the site from one that's community edited to one where the community only gets to suggest edits and we need moderators to approve every edit?

If we change the level at which users can see deleted posts we might also lose most community moderator oversight.

So I guess we'd look at how many people we would cut out from such activities and whether we'd be left with a viable population still able to participate in the way they have become used to doing.

I'd suggest working out how many people are currently at those key thresholds now, how many would still be at those thresholds post beta and also use their powers - badges such as Strunk and White may help here. Then post to your site's Meta and see what people's reactions are to the reduction in numbers.