11

Is there data about the amount of reviews that the new not-so-shiny review notification icon has caused? Has it increased the amount of reviewers? Decreased? No significant change?

8

To address this, I'm looking at the top 10 sites on the network by number of 10K users active in the past 30 days (by which I mean, users with the moderator tools privilege, not necessarily users with 10K reputation):

10K Site Name
250 Mathematics
128 Programmers
124 Super User
114 MathOverflow
110 Code Golf
103 English Language and Usage
99  Server Fault
98  TeX - LaTeX
92  Gaming
89  Code Review

I'm ignoring Stack Overflow because the indicator doesn't work the same way there. For what it's worth, there are more active 10K users on SO than on the rest of the network combined.

Here are graphs showing the number of active 10K reviewers going back to late February, with an arrow pointing to the first week with the new 10K review indicator:

10K reviewers by week

...and here are graphs for the same sites showing the number of reviews performed by 10K reviewers during the same time period:

10K reviews by week

As you can see, there's a noticeable bump in both reviews and reviewers when the change first rolled out, decreasing slowly at first and then falling off noticeably in June.

For comparison, here's the graph of reviewers less than 10K:

sub-10K reviewers by week

It's hard to draw any conclusions from this beyond "hawthorne effect"; both the number of 10K and sub-10K reviewers are seeing some heavy seasonal effects as folks head out on summer vacations and so on, but even ignoring that it's clear that not all sites saw much of a difference and for the ones that did it was at best a temporary bump.

To further obfuscate the potential effect here, it's entirely possible that a good many people on these sites aren't even seeing the indicator most of the time because there's nothing in any of the queues (or less than the threshold necessary to display the indicator). Mathematics, for instance, has 250 active 10K users but averages just over 62 active 10k reviewers per week - yet it has no real backlog to speak of, meaning a good many potential reviewers may not see the need to get involved. In contrast, Ask Ubuntu (not included in the graphs above) has only 66 active 10K users and averages a hair under 20 active 10K reviewers per week - a higher % of the total, but given the perpetual backlog clearly not enough to completely stem the tide.

I strongly suspect that for sites which always have a few items in one or another review queues, the appearance of the topbar indicator is too static to really serve the purpose it was intended for. Something along the lines of this proposal would probably serve us well in these cases.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .