A few weeks ago, we tested a modified review queue indicator for small sites. In essence, the top bar icon will get a red dot whenever there's any task and not just when the count reaches a discrete threshold. The experiment seems to have worked for small sites, but can be annoying when there are many active reviews. I'd like to come up with an objective criterion we can apply to determine which version to deploy to each site. (And, as sites grow, there will be a criterion to know when we ought to change the setting to the danger-lever version.)

I should note that:

1. People are no longer notified of review queue tasks that they can't access because of their privilege level and

2. People might still be notified of review tasks they have already completed if they don't get cleared after an hour

As a result, false positives are still possible for sites with few non-moderator reviewers but should be rare otherwise.

Proposed criterion: maximum median tasks per hour <= 2

The original design for the indicator was built to fire about 10% of the time. The notify-immediately version of the indicator will alert more often than that unless a site is extremely slow. But that's balanced by the benefit of being able to clear the review queues more often. If the typical hour on a site has only a few items to review, you might not mind getting notified every hour since you can clean them out each time. But if the typical hour has many more items than anyone can reasonably review, the alert is of the worst kind: constant and uninformative. By this criterion, here are the statistics for the experiment sites for the first two weeks of December 2017:

Site                       Queue           Median hourly tasks 
----                       --------------  ------------------- 
Meta Stack Exchange        Close Votes                       1
Movies & TV                Close Votes                       5
Internet of Things         Close Votes                       2
Martial Arts               Suggested Edit                    0
Role-Playing Games         Suggested Edit                    0
Worldbuilding              Close Votes                       2
Arts & Crafts              Suggested Edit                    1
Unix & Linux               Close Votes                       9
Computer Science Educators Close Votes                       1

Unix clearly has too many for the indicator to be helpful. (I ended the experiment for them last week.) Yes, they normally have few tasks outstanding because the community is very responsive. Warning sooner did reduce the time review tasks had to wait, but the satisfaction of completing the queue is ephemeral as new items arrive constantly. Meanwhile, there are hours in which nothing new comes to even RPG's most active queue. (The chart suggests it is Suggested Edits, but since the median is 0, it could be any of the other queues are more active. Depends on how SQL sorted the results.) So I think the median tasks for the most active queue is a pretty good metric.

As for the actual criterion, 2 seems reasonable to start with since:

1. It mostly divides the experiment sites between between those that appreciated the change and those that didn't and

2. It's easy to imagine someone doing a couple of tasks between answering questions and not feeling put out by the interruption

The caveats for those points are:

1. Movies & TV seems to like the new indicator and would fail this criterion and

2. This assumes that the other queues are basically empty and that reviews are not time consuming

Rollout process

I began rolling it out on sites with zero median tasks per hour. Assuming all goes well, we'll move on to the sites with a maximum of one per hour:

• Fitness and Nutrition
• Motor Vehicle Maintenance and Repair
• Parenting
• Aviation
• Poker
• Space Exploration
• Reverse Engineering
• Language Learning
• Hardware Recommendations
• Literature
• Politics
• ExpressionEngine
• Mythology
• Genealogy and Family History
• Arts & Crafts
• Embedded Systems
• Puzzling
• Vegetarianism
• Joomla
• Interpersonal Skills
• Mathematics Educators

What am I missing and what should I look for when the change goes to more sites?