With the rollout of the new top bar, you’ve probably noticed the new review indicator:
It’s possible there will be a different review icon by the time you read this. But this post is just about the red circle on the icon. You’ll see the indicator is turned on:
- if any review queue has a red dot (explained below) and
- if it’s been at least an hour since you clicked on the review icon.
So the idea is that when you see the indicator light up, it’s a reminder that there are reviews to be done if you haven’t looked recently. The goal is to draw reviewers’ attention to queues that can use some help.
Each queue may be in one of three states:
- No dot (empty queue)
- Grey dot (at least one but fewer than N review tasks)
- Red dot (N or more review tasks)
Each review queue has its own threshold (N). The default values are:
|Help (Stack Overflow)||150|
|Triage (Stack Overflow)||100|
We call more than N tasks the “danger zone”, but the thresholds aren’t tied to any particular system limitation. Instead they are based on the 90th percentile of hourly samples of pending tasks. To put it another way, we should expect each queue to be in the “danger zone” roughly 10 percent of the time. Obviously, each site has its own level of activity so we can (and do) adjust the thresholds from time to time. If we do that job right, it shouldn't matter what the exact numbers are.
How we adjust the thresholds (for now)
First, we group the queues based on privilege level:
- Close and Reopen
- Suggested Edits and Low Quality (+ Help & Improvement on Stack Overflow)
- Late Answers and First Posts (+ Triage on Stack Overflow)
For each group, we calculate the median and 90th percentile active reviewers per hour for the last two weeks. Then we calculate the median and 90th percentile pending reviews for each queue. The idea is to estimate how many people can work on each queue and how much work is available for them to do.
Then we run some checks of potential problems:
If the threshold is less than the 90th percentile of pending reviews tasks, that queue will likely trip the indicator more often than we'd like. In this case, raise the threshold. (We have a little fudge factor here to avoid making small changes each time we run the check. We've noticed that review queues tend toward the threshold we set. There's a ratchet effect in play here.)
If the threshold is more than twice the 90th percentile of pending reviews tasks, the queue may never trip the indicator. That doesn't mean these queues won't be cleared since people might do these review tasks without prompting from the top bar indicator or while they are doing other reviews. But we'd like all of the review queues to get attention from time to time.
reviewers warned of same tasks twice
Especially on sites with a small number of active reviewers, there's a chance tasks you'd acted on will still be in the queue after an hour when the indicator can turn on again. Ideally, we'd keep track of how many reviews each user has outstanding and only show the indicator if they have something to do. But in that case we might as well show everyone the number of review tasks they might be able to do, which brings us back to the performance/accuracy problem that got us here. So we fudge it by checking if the median number of reviewers is less than the median number of tasks needed to complete the reviews. In that case, we raise the threshold.
The opposite problem is when there are many potential reviewers getting notified about too few reviews. If the threshold is too low (we check if it's less than the 90th percentile of reviewers in an hour), there's a chance reviewers will visit the queue only to be locked out of the tasks. In this case, we raise the threshold.
Over the next few weeks, Shog9 and I will be looking a the queries he wrote to see if any queues across the network need their thresholds adjusted. Assuming we've hit upon appropriate criteria, we'll look into automating these adjustments.
As always, this is complicated by caching. You might see the site review indicator light up and then not see any of the individual queues in the red. In that case, it’s quite likely one or more of the queues just dipped under the threshold (and might pass it again soon).
Finally, all active tasks are counted for each queue when determining if a queue is in “danger”. Thus, there may be times when the indicator will light up but you’ll be unable to do anything about it—for example, if you’ve already done all the reviews you’re allowed to do for the current day in that queue, or if all the items to be reviewed are your own work.