Stack Exchange reputation leagues are now listing only the top 400 users

Question

Stack Exchange is currently in the process of migrating some of our backend infrastructure. Mostly, this migration comprises long-running behind-the-scenes work that doesn’t have an impact on the day-to-day functionality of the network. This post serves to earmark one small part of network functionality that is changing within the next week.

In case you’re not already aware, reputation leagues are an informal way of tracking reputation on stackexchange.com, and allows users to retrieve their relative position on the reputation highscores as compared to other users. We’re making a small change to the way we calculate and run the Stack Exchange reputation leagues page on the network site.

In the current state:

• Any user over 200 reputation receives a rank in the reputation leagues (about 800,000 users).
• Due to query timeouts, the data are often out of date or inaccurate.

Shortly after we deploy this change:

• Only the top 400 users will be listed on the reputation leagues per site and time period.
• Users who are not in the top 400 will no longer be able to view their rank in the leagues, but will be able to see how far they are from placing in the top 400 users.
• The data will stay up-to-date, accurate, and refresh on a predictable interval.

Answer 1

I put this on my resume (for math.se) and have gotten jobs because of it.

You’re destroying a valuable feature for higher rep but not top-400 users.

If performance is the concern you could reduce the frequency with which the reputation leagues are updated (ex: once a month).

Answer 2

I'm curious.

What actually made this feature be expensive to run?

How does this change make it less expensive?

I'm having a hard time imagining. To be able to rank anyone (accurately), don't you need to calculate the same information that you would need to rank everyone, since one's ranking is relative to everyone else?

What makes this expensive compared to the /users page? (presumably- since that tab is staying). Is it the ranking diff from the previous period? I can't understand why that alone would make such a difference. Wouldn't the amount of storage and calculation work be constant per user?

At least for the weekly league, it wasn't even updated live on each request. At least- not according to How often do the User Reputation Leagues weekly stats get updated?. If the daily one was updated live and that was the problem, just make it update at a fixed time each day.

Answer 3

I'd like to respectfully request StackOverflow to reconsider this change.

The decision to stop tracking reputation rank for most developers seems a bit heavy handed. Many developers value their rank on StackOverflow and it's a small important reward for the work they have put into helping the StackOverflow community.

As others have mentioned tracking the reputation of only the top 400 developers is far too narrow.

Take my case as an example. I've been ranking at about the top .7% both for this year and for all time. And yet, currently, I'm not ranked. That's crazy. I'm in the top 1% of developers on StackOverflow and I no longer get a ranking?! What?!

The ranking page says I need 35 points for the week to be tracked.

While I think that's far too high of a threshold (35 points for the year would be much more inclusive), I've earned 50 points for the week and still no ranking.

Update 1 day after Original Post

It's Wednesday. I now have 40 points for the week (see below) and 50 for last week (see prior screenshot), but I still have no rank on my profile activity page.

So tell me again, how is this new system supposed to work? Maybe the issue is that of the 40 this week, 20 are from StackOverflow and 20 are from StackExchange Meta.

Update 3 days after Original Post

It's Friday. I now have 40 reputation this week just on StackOverflow alone which come from 20 today and 20 yesterday. And I have 50 on StackOverflow alone the prior week.

And still, the profile shows no ranking.

One thing I learned is that the "Prior Week" shown in the menu is based on a sliding prior week rather then a cutoff day. So the prior week today is a different set of days than the prior week yesterday or the day before. I guess this makes perfect sense. But it makes it even harder to explain why there is no rank on the profile.

@slate Here are multiple screen grabs showing what the points menu currently looks like in my account. The new algorithm isn't providing a rank on the profile for people who received greater than 50 points in the last week.

Here is a link to my StackOverflow profile: https://stackoverflow.com/users/1415614/ronc

Recommendation

If previously any developer with 200 reputation received a rank (about 800,000 users) and that was more than the system could handle without timing out the query periodically, how about raising the threshold to 1000 reputation points?

This would still be a large inclusive pool of developers, and for new developers, the hill to climb to get to that level (1000 points) is somewhat reasonable.

Raising the threshold from 200 to 1000 points should greatly reduce the amount of work the query needs to do and hopefully allow it to consistently return results without timing out.

But why does it matter?

A developer's rank, in many cases, literally helps them get jobs.

Don't take that away from them.

Developers shouldn't need 30K+ reputation to get ranked.

Motivation Misunderstood: Why Developer Rank Should Matter to Stack Overflow

Today, October 31, 2024—about five months since my original post—I revisited Stack Overflow and noticed a "Welcome back, RonC" message alongside a new chart displaying my reputation growth and badge progress. It made me laugh; it seems like Stack Overflow is trying to re-engage users by highlighting their progress.

However, what management is missing is that rank is the true motivator for developers. Without a ranking system, developer participation will continue to decline.

Management couldn’t have picked a worse time to remove developer rank. On the consumption side, developers are increasingly turning to LLMs for programming questions, and now on the production side, Stack Overflow has just removed a key motivator for participation.

Answer 4

That is unfortunate.

At a minimum some effort could have been put into offering a SEDE query to replicate parts of the functionality.

For example this SEDE query takes the most relevant reputation contributors (mainly votes) over the last month, quarter or year and given your userid then ranks all users and shows (roughly estimated) your place in the ranking.

-- timeframe : Timeframe "yy for year, qq for quarter, mm for month"

declare @cutoff integer

select top 1 @cutoff = v.id
from votes v
where creationdate < dateadd(##timeframe?yy##, -1,  DATETRUNC(week, getdate()))
order by creationdate desc

select id, displayname
into #users
from users
where reputation > 200

create index #ix_users on #users(id)

;with user_scores as (
select u.id
     , sum(
       case  v.votetypeid 
         when 1 then 15
         when 2 then 10
         when 3 then -2
         when 9 then v.bountyamount
         else 0
       end
     ) [total_rep]
from #users u
inner join posts p on p.owneruserid = u.id and p.communityowneddate is NULL
inner join votes v on v.postid = p.id and votetypeid in (1,2,3,9)
where v.id > @cutoff
group by u.id
),
users_ranked as (
select rank() over(order by [total_rep] desc) [rank]
     , *
from user_scores
)


select 'You' 
     , [rank]
     , concat('site://users/', r.id, '|', u.displayname) [Link]
     , [total_rep]
from users_ranked r
inner join #users u on u.id = r.id
where r.id = ##userid##
union all
select 'Community'
     , [rank]
     , concat('site://users/', r.id, '|', u.displayname) [Link]
     , [total_rep]
from users_ranked r
inner join #users u on u.id = r.id
order by 1 desc, [total_rep] desc

This query runs within the limit of 2 minutes on SO by excluding all users with less than 200 reputation.

I suggest we iterate on this version to see how much juice we can squeeze out of it without running out of resources. For example given bounties and accepted suggested edits aren't accounted for yet. Feel free to fork the query and have a go on your own.

Keep in mind SEDE is updated once a week on Sunday.
Use the educational SEDE Tutorial written by the awesome Monica Cellio.
Say "Hi" in SEDE chat.

Answer 5

I'm very unhappy with this feature being cut. I always used this X% number from my user profile to quickly estimate the total rep I'm going to get this year.

• Due to query timeouts, the data are often out of date or inaccurate.

From what I saw, it used to update once a day on SO (along with some other stats, which I find appropriate considering the daily rep cap). Sure, such period is likely too big to accurately rank those in the very top of the list, but then this info will likely become outdated in a few seconds as well, so it's not really worth much accuracy anyway.
I don't remember it timing out and not being displayed at all (but I get that it's not calculated in real time, so this probably doesn't happen).

From my understanding, the biggest problem is to calculate the rep gain per period, but you still calculate them for all users! You only have to sort them and divide by user count in advance!
The All Time query linked below executes in 3 seconds (including the query itself), so that second part shouldn't be a problem.

From comments:

If you have a need to access analogous data, you may do so at your leisure using the Stack Exchange Data Explorer.

Are there any existing queries? I wasn't able to find anything except the All Time variant. Could you provide the one you used, so the community could use or try to improve it?
(this would be a great idea for an extension/​script if someone is willing to create it)

It was also not working as designed regardless, [...]

P.S.: You know what's really not working? The "people reached" thing when you write late answers.

Answer 6

"I'm a little upset" by this.

If there are some query timeouts, lot's of data (800k is a lot?), etc., can't you either:

• include users with 500 (or 1000) reputation points in the statistics, instead of 200? This will greatly reduce the amount of data.

• leave "Top % this year/month/overall" (maybe exclude week and day) on the profile page, and leave query feature for just top 400?

Answer 7

bug design profile-page reputation-leagues

OK, here's another regression if we're talking about bugs here. When you're not ranked you get an award for nothing:

Answer 8

I finally made it to the top 1% of users with 200+ reputation on Cross Validated, and then that number disappeared. My last remaining motivation to post on Cross Validated disappeared along with it. The breakdown of how many users there are with which reputations also disappeared, so what use is it to know that I am #122? What does that mean? The baseline for comparison has disappeared. For example #122 of 400 does not mean much. Answer this: So what?

Software acceleration would most likely solve the problem. Why was this not done? The real issue for those who are ranked is how many total users there are and how many of those are expert enough users to be considered contributors. More than 200 for a reputation doesn't quite do that. I seriously suggest that you find something to provide feedback to users lest they abandon the site for greener pastures.

The prior Leagues method of ranking those with +200 reputations was flawed. The current Leagues of listing only the top 400 in the "Leagues" is even more flawed. For example, on a site with 400 users, they would all be listed, and on a site with 40,000,000 users almost no one would be listed. Thus, the current listing is useless. This problem could have been discussed with the users of Cross Validated and would have been by a more open minded administration. The attitude of "us" versus "them" is mechanistically predatory and is harming the quality of StackExchange (SE) decisions.

Consider, for example, that one can quickly calculate total reputation on a site, and the total number of users on the site. From that, one can calculate what the average reputation is on a site. So suppose that average is a reputation of 101. Now suppose, that an individual has a reputation of 10100. Then that individual has as much reputation as the mean reputation of 100 people. From that factor of 100 times, one looks up the level of expertise in a established lookup table. One then finds that 100 times the average reputation of our fictitious individual corresponds to that of, let use assume, a moderately advanced expert. All the implementation of that would require is to take a site, lookup who the professors, post-docs, undergraduates etc. are and from that build up a typical lookup table. The advantage of doing that is that unlike what is being done now, a basis for comparison that has some cross site validity has been established. The overhead for doing that is minuscule. The overhead for a ranking system, as is currently used is tedious, and the rankings are not very useful because the number of users who make a significant contribution is not similar between sites. The administration has not put a lot of thought into what reputation means. Think first, act later, and don't try to centralize thinking into a handful of employees to satisfy a few million users. That type of top down or communistic "central committee" approach is so top heavy as to be untenable for large scale management. It is, for example, why the Soviet system failed. One can run very large organizations, but one only does so if respect for natural law is maintained, that is, by delegating authority to those lowest levels at which the decisions are most naturally implemented. I seriously doubt that SE will consider proper management skills as that appears to be culturally unachievable given current SE corporate structure. Rather than repeating the same mistakes that led to the failure of the Tower of Babel, SE would be better advised to get religion. Seriously, SE would be better advised to make better use of the resources at their disposal.

Answer 9

If not a full reimplementation, please add back "top n%" stats for months, years, and all-time.

The statistics powered by the reputation leagues was an inexpensive incentive for both regular contributors (whose posts provide a positive asker experience) and new signups/contributors (who the company is actively trying to court). I think that the removal of these statistics is a significant negative for the community—devaluing prolific users' contributions and eliminating near-instant gratification for new users—but a bucket-oriented reimagining of the feature could make it both better and cheaper-to-run than it used to be.

I include in this the tag top users page linked under the descriptions on tag pages; the top user statistics are currently broken on SE. Perhaps that's just a bug, but I imagine it could easily be an unmigrated expensive query.

---

It might not be very intrinsically valuable to have a quickly-updating "league" where day-by-day a person can see whether they are #2975 versus #2973 (for all but the most competitive of contributors), but the leagues and stats did have a significantly valuable role: it provided some nice statistics that could translate their reputation into forms that non-enthusiasts could understand. As the reputation page still describes it:

Reputation is a rough measurement of how much the community trusts you; it is earned by convincing your peers that you know what you’re talking about.

Stack Exchange's status as a well-known public utility gives it some prestige, and its sense of reputation is a passive motivating reward for contributors with real-world résumé benefits. However, reputation has always been an arbitrary number calculated from votes received, (down)votes cast, bounties, and accepted answers such that one might have some difficulty conveying the meaning of a value like "49670 reputation" or "18 gold badges" to hiring managers, book publishers, colleagues, or others who might not know or care about the details of the reputation calculation.

Compared to absolute rep values or even full orderings, "top 10% in 2023" or "top 3% in May 2024" is much easier to understand. For the tag top users, one could likewise imagine being a "top 5% all-time contributor to Android questions and answers" to be of similar value: Even gold badges don't inherently describe their value or scarcity, and are difficult to earn on low-traffic tags where the value of niche expertise could be the highest. Statistics like "top n%" are self-describing and inclusive, gaining relative scale even if they lack precision compared to "#n", and they can also provide some immediate gratification for new contributors: you can still be a top 10% contributor for a given month or tag even if you happen to be #401 in rank.

To be mindful of cost, one could calculate only one significant figure (top 0.1% to 0.9%, 1% to 9%, 10% to 50%): for the purpose of communication, 27% ≈ 28% but 7% ≉ 8%, and that limits the number of percentiles to calculate to a pretty manageable list. Likewise, because this is intended to reflect contributions over time, this query could be run weekly or monthly while still providing its intended value. Finally, this is quite quick and cheap to calculate: each of those distribution tables is about 23 small integers—enough to fit on a cocktail napkin—and tools like PERCENTILE_CONT make it very cheap to calculate or approximate natively even on sharded or sampled data sources.

Please consider a reimplementation here: this might all be for imaginary internet points, but you had a pretty fast and compelling way to make reputation points meaningful outside of Stack Exchange, and I think it would be valuable for contributors and company both to see them come back.

Answer 10

I enjoyed every 1% increase that happened and ignored the meaningless raw rep score.

Just like figuring out what percentile a person's height is doesn't require measuring the entire population, get a random sampling, and figure out what roughly is each 1% increment's average is, then you have 100 values, and for the user page, just figure out what the closest value is for that user.

It doesn't have to be super accurate, and it doesn't have to update more frequently then once a week. Any variability up or down due to statistical error would really just make people more competitive and try harder to boost their score.

Answer 11

There seems to be a bug after this change. I have never reached 28k rep on Mathematics so I'm not sure where this surplus comes from.

Answer 12

Way too many users get ranked, just because they reached 200 rep and got an association bonus.

They are all slightly above 300 rep and have +110 weekly change. +10 from a upvote that pushed them past 200 rep and +100 from the association bonus.

Almost all ranks in the last quarter (#300-#400) are occupied by them.

According to this meta post, association bonus should have not have been counted.

Answer 13

bug

When I'm clicking on the '#3 overall' on my profile page:

I end up on this page which says I'm 'not currently ranked':

Clicking on the 'All Time' tab resets things back to normal, and I'm ranked again:

Answer 14

feature-request status-completed

Choosing to see 50 users per page triggers CloudFlare's "too many requests" and cause all profile pictures hosted in SE to be blank.

This doesn't look nice:

I suggest removing the option to see 50 users per page, perhaps the 30 as well, leaving it only 15 per page.

With only 400 total users available to page through, it's no big deal.

Note: actually, it's even worse: this cause the block to be network-wide, e.g. now I can't even see the images I upload here. Another option is to add a warning on the reputatuion league page that it might block access to images when viewed.

Answer 15

bug

Not sure if it's a bug, but at least it looks inconsistent. If you've got no rep per period, the information about your ranking is not displayed at all (not even the "rep needed"):

Edit:
And the "month reputation" actually doesn't match with the audit page. This is because the latter doesn't account the rep lost due to "User was removed".

Not sure which one is correct here since technically I didn't lost this reputation this month, but rather didn't get it in the first place...

Answer 16

This is a really bad idea and undermines the whole reasoning behind a reputation system.

For instance, what an outsider gets from hearing about a 10k reputation on Stack Overflow? Bare with me: without context, 10k could be common or rare, it could be easy or hard, and therefore it conveys no information -- it's just a big number. And a placement in a ranking is the way to give it meaning and context.

To know/show how well one fares among a group is the core purpose of a reputation system. By removing ranking, we lose the context that gave points its meaning. Our contributions to the community become just points. It defeats the reputation part.