20

SE site graduation used to involve a bunch of things happening at the same time: removal of the "beta" label, a shiny new design, privilege thresholds increasing from beta levels, inaugural moderator elections, and a few other bits and pieces. Since design-independent graduation kicked off in 2015, these different facets of graduation have become increasingly disconnected from each other, and when graduated sites get their designs and rep threshold changes can vary widely.

In the last couple of years, a vast number of SE sites have quite suddenly "graduated" (i.e. had their beta label removed, and in some cases no other changes), leaving very few beta sites remaining:

Many of these sites, and earlier graduated ones too, are still lacking other features of graduation. Obviously, getting a design depends on SE designer time, and getting mod elections depends on a bunch of factors including CM time, but changing reputation thresholds is presumably (from SE's point of view) just a matter of a simple button push, which could be requested by the community on meta and implemented with less delay.

The purpose of this post is to figure out guidelines for when a site/community should request this. Some pros and cons are listed in the PPCG meta that I linked earlier. This is an important part of graduation and could help a site to feel "mature", much like the beta label removal, and would (at least in theory) restrict participation in moderation activities like closing/deleting to those with more experience and knowledge of the site. On the other hand, it would drastically reduce the number of people able to perform those activities, which for some smaller graduated sites could be crippling.

What metrics or statistics would be useful to check? Number of users (or of active users, by some definition of "active") currently above certain rep thresholds? Number of reviews and active reviewers per year? Percentage of questions getting closed and answers getting deleted? Anything else?

3 Answers 3

16

This is something that the CM team has been thinking about since before I pushed that first set of 29 sites out of beta in 2019. It was, in fact, part of our conversation when we made the decision to do that. One of the things that we've noticed and reiterated nearly every time we've made changes to the Graduation process is that we recognize that there's not one "right" site size and that not every site will be the size of Stack Overflow.

That's no less true today than it was in 2015 when Ana wrote Graduation, site closure, and a clearer outlook on the health of SE sites or I announced those 29 sites leaving beta in 2019 or Rosie announced the round of 59 sites at the end of 2021.

You recognized the same thing that we did, though - we have more and more sites out of beta and some of those sites are even getting site designs… and we asked ourselves… "Where do the 'graduated' or 'full' reputation privileges come in now?" It's not happening when they leave beta… it's not happening when they get a design… so when?

We asked ourselves the same questions you're asking - what metrics would we outline for when we would reach out to a site and say, "Congrats, it's time to have higher reputation levels!" or when a site could come to us and ask "We can haz higher rep levels please?" And the answer that we came to is - if it's not broken, let's not "fix" it. That is to say…

We have no plans to change reputation levels across the board on a site from "beta" to "full" ever. We will only adjust individual reputation levels if the current number is not working.

Now, whether this decision excites you or makes you furious, let's take a minute and work through why we've made this decision.

Site age != big community != lots of rep

When sites moved through the process more quickly, there was often more content being created, more votes being cast and more people with the reputation needed to get the higher reputation levels. But as sites have focused on smaller and more niche subjects with fewer experts to participate and judge content, we realized that those "full" rep levels might never be a good fit, just as getting to 10 questions per day might never be possible. But - that doesn't mean the community isn't sound, healthy, and serving a valid purpose.

Since the last time we gave people higher rep levels, I can only think of one site that's asked for them, maybe two… and I can guess why that is.

Losing privileges is a bummer.

Losing privileges is frustrating to people. I see people complaining about it often - paying for bounties causes them to lose editing privileges or posting an unpopular answer drops them below the reputation needed to see deleted posts or downvoting loses them the privilege to downvote (probably my favorite)... It's just not fun for anyone involved. Heck, it's why we don't take away badges (in most cases) even if you would lose them otherwise.

A bunch of people losing a whole raft of privileges because a subset of users decided it was time to require higher privileges on a site… just doesn't make for great community happiness and can, actually, lead to poorer community engagement. Particularly in cases where there really weren't enough people with the privileges to do tasks that required several participants.

Honestly, I feel like a big part of why we tested 3-vote closure (and it's been so popular on sites) is because many of those big sites with higher rep levels don't have enough people with the close privilege who are interested in participating in that task. Imagine doing that to smaller sites that only have 5-10 active people with 3k reputation. The people who lose the privilege aren't necessarily going to be motivated to earn it back, and may just wander off.

Why are we still using reputation to determine access to privileges anyway?

If you read much of what I post on meta, you may have heard me opine about how we need to stop locking privileges to reputation, anyway. Someone with 100k reputation doesn't necessarily know how to edit a post well but someone with 500 reputation who's had their last 100 suggestions accepted probably does. In a perfect world, we would much rather give access to tools to people who have shown through some method that they know what they're doing - and make sure they continue to perform well at those tasks, lest they lose that privilege.

While we don't have it on our roadmap to start this work, right now we don't think it makes sense to further push the rhetoric of reputation being equivalent to ability on these sites by making people re-earn privileges they've already shown they're more than capable of using well.

Our site has an influx of crappy tags from low-rep users… we need higher privilege levels!

Whether it's tags or close votes or post deletion, this is where the second half of the big bold header up top comes in. "We will only adjust individual reputation levels if the current number is not working." So, the narrative here isn't "you're just going to have to deal"; it's - "OK, tell us about the problem you're facing". If your site has newer users going on tag creation sprees or voting to leave open questions that should be closed, that's bad! Let's see what we can do about it - and this includes cases where the "full" privilege levels may be too high.

Whatever the privilege, we can adjust it on a per-site basis without changing the entire site to the full site privilege levels. This is why Stack Overflow has even higher levels for some things - including tag creation. We want to target the specific problem your community is facing rather than having potentially negative impacts on all privileges because of an issue with one or two.

While I can't say we'll never change sites from beta to full rep levels in the future, sites should be aware that it's unlikely we will. If closing and reopening are working fine at 500, what's the purpose of flipping it to 3k? If there is an issue, we can address it; otherwise, let people do their thing!

For a full listing of the privileges and exceptions, check out What are the reputation requirements for privileges on sites, and how do they differ per site?

9
  • 1
    Thank you very much for the official answer! I wasn't sure if this post was ever really on the company's radar, so I'm glad that you (singular or plural) have put thought into it. I'm accepting this answer to pin it to the top, although I'm not sure if it fully answers what I was originally asking ... [further comment incoming] Feb 27, 2023 at 14:17
  • One request, to fully address my original question (although I realise this might not be something you can meaningfully advise on in general): any tips for recently graduated sites on how to decide when/whether to ask for privilege threshold raises? Apart from new users going on tag creation sprees? Do you see it as an "only request threshold changes if it's necessary to address some specific site problem" thing, or could a site request changes just because the community feels it's mature enough / has enough high-rep users? Feb 27, 2023 at 14:18
  • 3
    Right now, I just don't know that I see any sites in a state where they really need higher rep levels. I'd kinda throw the question back at you - why does the site being "mature" indicate a need for the full rep levels at all? If closing and reopening are working fine at 500, what's the purpose of flipping it to 3k? If there is an issue, we can address it! :)
    – Catija
    Feb 27, 2023 at 14:31
  • I guess a good case for bumping the privilege levels up to full would be if the "everyone's super so no one is" problem occurs (so many people have higher privileges such that many people are abusing them). That said, I do agree that the reason full levels were deployed in the past was because it was considered a "milestone" rather than because it was actually necessary for the site. I've updated the privileges FAQ to link to this answer. Feb 27, 2023 at 19:04
  • 1
    @sonic ... I mean, I don't think that analogy really works here. I'd argue that our format is designed (even if it doesn't do it well) to grant access to people who understand the tools and how to use them appropriately (as agreed upon by the community). If everyone could do that, there'd be no issues with everyone having the tools. That might even be seen as the goal of a site. As with any situation, the solution needs to fit the problem. High-rep users can abuse privileges, too. Changing the rep to close from 500 to 3k only solves the problem if the people voting incorrectly have <3k rep.
    – Catija
    Feb 27, 2023 at 19:13
  • I was referring more to the case where a site has around one third to one half of its active population have the 4k trusted user privilege. In that case, with such a high population of users with moderation capabilities like voting to delete or viewing deleted posts, it could effectively become an elitist group, and the chance that one or more users will abuse those privileges (say, to stir up trouble when a mod has wisely removed a problematic post) is higher. I'd say that's a good case to have increased separation between privileges. Feb 27, 2023 at 19:22
  • 1
    @sonic "Let's see what we can do about it - and this includes cases where the "full" privilege levels may be too high."
    – Catija
    Feb 28, 2023 at 17:34
  • Ah, the way I'd previously interpreted it was "the full privilege level may be too high [to be deployed on a site that currently has beta levels]", as in a site where 1000-1500 rep users don't correctly review edits but 1500+ rep users do review correctly, so the full site bar of 2000 would be too high to deploy on the site. Still, though, it would be nice if an answer were posted there making this explicit. Feb 28, 2023 at 20:21
5

What metrics or statistics would be useful to check? Number of users (or of active users, by some definition of "active") currently above certain rep thresholds? Number of reviews and active reviewers per year? Percentage of questions getting closed and answers getting deleted? Anything else?

What you're describing seems to me to be a very global look at the site.

I would suggest looking at each privilege individually, consider who uses that privilege in the current situation and then see how they would be affected by the new situation.

Let's take close vote reviews as a practical example. I would look at all the close vote reviewers over the past year. Then make a list (or bar chart) and sort them in descending order of how many close vote reviews they did over that period. So at the start we get the most active reviewers while the end consists of a tail of users who completed only a few reviews that year.

In my head it looks like this power law graph with the most frequent reviewers on the left and the least frequent ones on the right. I don't know if it actually looks like that, but online participation (like many other data sets in social sciences) tend to follow such a distribution.

Let's say we color all the bars blue indicating the corresponding users are able to review at the current rep levels. If we increase the rep requirement, some of those users won't be able to review. In my imagination we'd have a slider in which we can virtually increase the reputation requirement and those users who fall off would turn red. With that, you should be able to see if a proposed increase leaves enough recently active reviewers to keep the site running. How many active reviewers may lose their privilege for the change to be successful is up for debate, but I think such a visualization is a good option to get the practical implication in one graph.

Another factor in play is how many active reviewers are actually available. If there are only 6 active reviewers then losing 2 might be a big burden on the site because it hampers the ability to close questions. If you had 60 active reviewers then losing 20 (perhaps only temporarily) wouldn't be as bad.

One downside of this approach is that it only considers the current user base. With beta rep requirements, new users can get into reviewing after a reasonable bit of participation. With graduated reputation requirements it will take much longer for new users to be allowed to review. That's an aspect to consider, but the bar chart approach doesn't account for that.

Another thing to consider is that some of those users who lose their privilege due to the reputation requirement change may be close to regaining the privilege. So you could make another progress bar chart to show for those who would be cut off, how far they are toward regaining the privilege. This could all be incorporated in the same graph by using some color gradient for those who lose the privilege.

3
  • Yep, I'm deliberately trying to take a global look. The ideal would be a relatively simple statistic (could be a combination of a few different figures, but not something enormously complicated or subjective) that could be used as a litmus test for whether or not a site is ready to get the higher rep thresholds. Then each site can figure out if they can request it already or what they need to do to reach that point. Sep 3, 2022 at 16:18
  • Think of the latest criteria for graduation. It's not just a one-size-fits-all statistic like "10 QPD" that they used in 2015, but it's based on a few different factors that together give a meaningful idea of a site's maturity/readiness. Something like that. Maybe "number of 10k+ users performing any site activity in the last 6 months" combined with "number of 3k+ users taking actions in the review queues in the last month" combined with "percentage of reviews performed by 3k+ rep users". Random examples, but you get the idea. Sep 3, 2022 at 16:21
  • 1
    @Randal'Thor In that case I'd go with my approach in list form and try to do that using SEDE queries. I'd still stick with evaluating privileges separately, but I don't think it's necessary to look at all of them. I think you need to look at frequency of using the privilege as well, if it's a relatively small group of active reviewers then you'd need enough of them to be available consistently. For example, a 10K user performing 5 reviews over 3 months compared to one participating in over 25% of the total reviews over that time period is a big difference, I'd say.
    – JJJ
    Sep 3, 2022 at 16:27
3

For a site to be in good health we really want a significant part of the moderation such as reviewing and editing to be handled by non-moderators. If we don't have that then we really don't have community participation, we just have moderator participation and they close and reopen by fiat.

So we'd need to see what effect changing the close/reopen privilege would have. How many of the existing close/reopen voters would we lose? Are those that would remain, active reviewers? Would we end up having the same very few people close and reopen everything?

Similarly with editing, would we turn the site from one that's community edited to one where the community only gets to suggest edits and we need moderators to approve every edit?

If we change the level at which users can see deleted posts we might also lose most community moderator oversight.

So I guess we'd look at how many people we would cut out from such activities and whether we'd be left with a viable population still able to participate in the way they have become used to doing.

I'd suggest working out how many people are currently at those key thresholds now, how many would still be at those thresholds post beta and also use their powers - badges such as Strunk and White may help here. Then post to your site's Meta and see what people's reactions are to the reduction in numbers.

2
  • Yep, this is what we need, but any suggestions on statistics for measuring it? Sep 3, 2022 at 16:23
  • 1
    Like a hard number for the minimum remaining number of close voters? Perhaps that's something for the site to decide for itself. Sep 3, 2022 at 16:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .