A few weeks ago, we tested a modified review queue indicator for small sites. In essence, the top bar icon will get a red dot whenever there's any task and not just when the count reaches a discrete threshold. The experiment seems to have worked for small sites, but can be annoying when there are many active reviews. With 170 sites, it's not practical to ask each meta to give us their preference. So I'd like to come up with an objective criteria we can apply to determine which version to deploy.

I should note that:

1. people are no longer notified of review queue tasks that they can't access because of their privilege level and
2. people might still be notified of review tasks they have already completed if they don't get cleared after an hour.

As a result, false positives are still possible for sites with few non-moderator reviewers but should be rare otherwise.

Proposed criteria: maximum median tasks per hour <= 2

The original design for the indictor was built to fire about 10% of the time. The notify-immediately version of the indicator will alert more often than that unless a site is extremely slow. But that's balanced by the benefit of being able to clear the review queues more often. If the typical hour on a site has only a few items to review, you might not mind getting notified every hour since you can clean them out each time. But if the typical hour has many more items than anyone can reasonably review, the alert is of the worst kind: constant and uninformative. By this criteria, here are the statistics for the experiment sites for the first two weeks of December 2017:

Site                       Queue           Median hourly tasks 
----                       --------------  ------------------- 
Meta Stack Exchange        Close Votes                       1
Movies & TV                Close Votes                       5
Internet of Things         Close Votes                       2
Martial Arts               Suggested Edit                    0
Role-Playing Games         Suggested Edit                    0
Worldbuilding              Close Votes                       2
Arts & Crafts              Suggested Edit                    1
Unix & Linux               Close Votes                       9
Computer Science Educators Close Votes                       1

Unix clearly has too many for the indicator to be helpful. (I ended the experiment for them last week.) Yes, they normally have few tasks outstanding because the community is very responsive. Warning sooner did reduce the time review tasks had to wait, but the satisfaction of completing the queue is ephemeral as new items arrive constantly. Meanwhile, there are hours in which nothing new comes to even RPG's most active queue. (The chart suggests it's Suggested Edits, but since the median is 0, it could be any of the other queues are more active. Depends on how SQL sorted the results.) So I think the median tasks for the most active queue is a pretty good metric.

As for the actual criteria, 2 seems reasonable to start with since:

1. it mostly divides the experiment sites between between those that appreciated the change and those that didn't and
2. it's easy to imagine someone doing a couple of tasks between answering questions and not feeling put out by the interruption.

The caveats for those points are:

1. Movies & TV seems to like the new indicator and would fail this criteria and
2. this assumes that the other queues are basically empty and that reviews are not time consuming.

Proposed process

My suggestion is to talk about the criteria for about a week and, if there are not better ideas, start rolling the change out to sites that meet the criteria next week. I'll start with sites that have median reviews in an hour of zero:

• Science Fiction & Fantasy
• Programming Puzzles & Code Golf
• Webmasters
• Retrocomputing
• Role-playing Games
• Board & Card Games
• Vi & Vim
• Community Building
• Sitecore
• SharePoint
• Italian Language
• Sustainable Living
• Anime & Manga
• Robotics
• Spanish Language & Usage
• Martial Arts

Assuming those go well, we can do the next two tiers in the coming weeks.

What am I missing and what should I look for when the change goes to more sites?