What criteria should we use to determine which review queue indicator a site will have?

Question

A few weeks ago, we tested a modified review queue indicator for small sites. In essence, the top bar icon will get a red dot whenever there's any task and not just when the count reaches a discrete threshold. The experiment seems to have worked for small sites, but can be annoying when there are many active reviews. I'd like to come up with an objective criterion we can apply to determine which version to deploy to each site. (And, as sites grow, there will be a criterion to know when we ought to change the setting to the danger-lever version.)

I should note that:

1. People are no longer notified of review queue tasks that they can't access because of their privilege level and

2. People might still be notified of review tasks they have already completed if they don't get cleared after an hour

As a result, false positives are still possible for sites with few non-moderator reviewers but should be rare otherwise.

Proposed criterion: maximum median tasks per hour <= 2

The original design for the indicator was built to fire about 10% of the time. The notify-immediately version of the indicator will alert more often than that unless a site is extremely slow. But that's balanced by the benefit of being able to clear the review queues more often. If the typical hour on a site has only a few items to review, you might not mind getting notified every hour since you can clean them out each time. But if the typical hour has many more items than anyone can reasonably review, the alert is of the worst kind: constant and uninformative. By this criterion, here are the statistics for the experiment sites for the first two weeks of December 2017:

Site                       Queue           Median hourly tasks 
----                       --------------  ------------------- 
Meta Stack Exchange        Close Votes                       1
Movies & TV                Close Votes                       5
Internet of Things         Close Votes                       2
Martial Arts               Suggested Edit                    0
Role-Playing Games         Suggested Edit                    0
Worldbuilding              Close Votes                       2
Arts & Crafts              Suggested Edit                    1
Unix & Linux               Close Votes                       9
Computer Science Educators Close Votes                       1

Unix clearly has too many for the indicator to be helpful. (I ended the experiment for them last week.) Yes, they normally have few tasks outstanding because the community is very responsive. Warning sooner did reduce the time review tasks had to wait, but the satisfaction of completing the queue is ephemeral as new items arrive constantly. Meanwhile, there are hours in which nothing new comes to even RPG's most active queue. (The chart suggests it is Suggested Edits, but since the median is 0, it could be any of the other queues are more active. Depends on how SQL sorted the results.) So I think the median tasks for the most active queue is a pretty good metric.

As for the actual criterion, 2 seems reasonable to start with since:

1. It mostly divides the experiment sites between between those that appreciated the change and those that didn't and

2. It's easy to imagine someone doing a couple of tasks between answering questions and not feeling put out by the interruption

The caveats for those points are:

1. Movies & TV seems to like the new indicator and would fail this criterion and

2. This assumes that the other queues are basically empty and that reviews are not time consuming

Rollout process

I began rolling it out on sites with zero median tasks per hour. Assuming all goes well, we'll move on to the sites with a maximum of one per hour:

• Fitness and Nutrition
• Motor Vehicle Maintenance and Repair
• Parenting
• Aviation
• Poker
• Space Exploration
• Reverse Engineering
• Language Learning
• Hardware Recommendations
• Literature
• Politics
• ExpressionEngine
• Mythology
• Genealogy and Family History
• Arts & Crafts
• Embedded Systems
• Puzzling
• Vegetarianism
• Joomla
• Interpersonal Skills
• Mathematics Educators

What am I missing and what should I look for when the change goes to more sites?

Answer 1

I think that this statement:

people might still be notified of review tasks they have already completed if they don't get cleared after an hour.

is a bigger issue than you are allowing for.

Why do we care about false positives?

On a particular small site that I am an active reviewer on, the close vote queue can sit with active items for entirely too long. Given the number of votes it takes to close items, this is not surprising in a smaller community. One solution for this problem, is for the moderator to come through and impart their will for the entire community, by processing the queue. While this will clear the queue, it will not allow the site to build a community of reviewers as the site tries to move from small to (hopefully) larger.

Therefore, the close vote queue will routinely not be cleared in an hour. So, what is the result? The indicator is then frequently on for those who have gone through the queue recently (last day or two). So, those who might actually be willing to go to the queue when needed, will get trained to ignore the indicator because it is too often wrong.

I realize that SO regulars will laugh at the notion that the close vote queue could possibly be cleared in a timely manner, but... it is possible. Unix, where I have been an active reviewer, and who was featured in your examples, manages to get their close votes processed on a regular basis.

Answer to your question:

• Yes, Please

  On sites with few reviews per hour, notify about anything to do in the review queues. Less than 2 per hour seems like a reasonable threshold.

• No, Thank you

  Please don't lie about there being something to do in the queues.

Answer 2

I just changed the setting on the following sites:

• Science Fiction & Fantasy
• Programming Puzzles & Code Golf
• Webmasters
• Retrocomputing
• Role-playing Games
• Board & Card Games
• Vi & Vim
• Community Building
• Sitecore
• SharePoint
• Italian Language
• Sustainable Living
• Anime & Manga
• Robotics
• Spanish Language & Usage
• Martial Arts

Feb. 15 update. I've expanded the list to the following sites:

• Fitness and Nutrition
• Motor Vehicle Maintenance and Repair
• Parenting
• Aviation
• Poker
• Space Exploration
• Reverse Engineering
• Language Learning
• Hardware Recommendations
• Literature
• Politics
• ExpressionEngine
• Mythology
• Genealogy and Family History
• Arts & Crafts
• Embedded Systems
• Puzzling
• Vegetarianism
• Joomla
• Interpersonal Skills
• Mathematics Educators

April 20, 2018: Added The Great Outdoors.

If you notice any problems, please post here on meta.

Assuming all goes well, we can do the next two tiers in the coming weeks.

Answer 3

Suggestion: Can per-site metas (with the possible exception of Meta Stack Overflow) get the quick indicator (i.e., red as soon as there is a review task)?

Rationale: Review tasks on per-site metas are very rare and get little visibility (thus taking forever). A quick review indicator could help.

Answer 4

Can't we make this a per site user setting?

Is there any reason not to? It could be just a checkbox on the review queue overview page.

☐ Show review queue indicator for this site.

Answer 5

I would like to have a traffic light indicator (or something alike to consider color vision impairment) for each queue to see what is more urgent, and one general indicator for the site. This is similar to the proposed change by dessert.

I don't know how if the time frame is correct, maybe it needs to be (automatically) adjusted for each site by considering the number of possible reviewers and the actual review work and the number of reviewers.

Thresholds for each queue:

• grey: any action available
• orange: a quarter of (new?) actions (of the previous week?) accumulated
• red: half (new?) actions (of the previous week?) accumulated

General light for all the queues:

• orange/grey: one quarter or more of the queues (available to the user) are in orange
• red: half or more of the queues (available to the user) are in orange

I think this would help to send a signal to users who don't go often to the review queue, but would find some queues to work in. It is not fixed to a certain amount, so for sites that work well the general indicator wouldn't bother users not interested in doing review, unless the usual reviewers can't handle the workload. Is quite similar to how it actually works (so maybe it isn't worth much or should the timing frame be one hour instead of quarters of the previous week).

This will probably mitigate the effect of new reviewers fading away after a change on the notification system because the notifications will depend on the community work itself.