How does the review-needed indicator work exactly?

Question

With the rollout of the new top bar, you’ve probably noticed the new review indicator:

or

It’s possible there will be a different review icon by the time you read this. But this post is just about the red circle on the icon. You’ll see the indicator is turned on:

1. if any review queue has a red dot (explained below) and
2. if it’s been at least an hour since you clicked on the review icon.

So the idea is that when you see the indicator light up, it’s a reminder that there are reviews to be done if you haven’t looked recently. The goal is to draw reviewers’ attention to queues that can use some help.

Each queue may be in one of three states:

1. No dot (empty queue)
2. Grey dot (at least one but fewer than N review tasks)
3. Red dot (N or more review tasks)

Each review queue has its own threshold (N). The default values are:

Queue	Default
Close	20
Reopen	5
Low Quality	4
Suggested Edits	3
First Posts	10
Late Answers	6
Help (Stack Overflow)	150
Triage (Stack Overflow)	100

We call more than N tasks the “danger zone”, but the thresholds aren’t tied to any particular system limitation. Instead they are based on the 90^th percentile of hourly samples of pending tasks. To put it another way, we should expect each queue to be in the “danger zone” roughly 10 percent of the time. Obviously, each site has its own level of activity so we can (and do) adjust the thresholds from time to time. If we do that job right, it shouldn't matter what the exact numbers are.

How we adjust the thresholds (for now)

First, we group the queues based on privilege level:

• Close and Reopen
• Suggested Edits and Low Quality (+ Help & Improvement on Stack Overflow)
• Late Answers and First Posts (+ Triage on Stack Overflow)

For each group, we calculate the median and 90^th percentile active reviewers per hour for the last two weeks. Then we calculate the median and 90^th percentile pending reviews for each queue. The idea is to estimate how many people can work on each queue and how much work is available for them to do.

Then we run some checks of potential problems:

1. over-warning

   If the threshold is less than the 90^th percentile of pending reviews tasks, that queue will likely trip the indicator more often than we'd like. In this case, raise the threshold. (We have a little fudge factor here to avoid making small changes each time we run the check. We've noticed that review queues tend toward the threshold we set. There's a ratchet effect in play here.)

2. never-warning

   If the threshold is more than twice the 90^th percentile of pending reviews tasks, the queue may never trip the indicator. That doesn't mean these queues won't be cleared since people might do these review tasks without prompting from the top bar indicator or while they are doing other reviews. But we'd like all of the review queues to get attention from time to time.

3. reviewers warned of same tasks twice

   Especially on sites with a small number of active reviewers, there's a chance tasks you'd acted on will still be in the queue after an hour when the indicator can turn on again. Ideally, we'd keep track of how many reviews each user has outstanding and only show the indicator if they have something to do. But in that case we might as well show everyone the number of review tasks they might be able to do, which brings us back to the performance/accuracy problem that got us here. So we fudge it by checking if the median number of reviewers is less than the median number of tasks needed to complete the reviews. In that case, we raise the threshold.

4. colliding reviewers

   The opposite problem is when there are many potential reviewers getting notified about too few reviews. If the threshold is too low (we check if it's less than the 90^th percentile of reviewers in an hour), there's a chance reviewers will visit the queue only to be locked out of the tasks. In this case, we raise the threshold.

Over the next few weeks, Shog9 and I will be looking a the queries he wrote to see if any queues across the network need their thresholds adjusted. Assuming we've hit upon appropriate criteria, we'll look into automating these adjustments.

Caveats

As always, this is complicated by caching. You might see the site review indicator light up and then not see any of the individual queues in the red. In that case, it’s quite likely one or more of the queues just dipped under the threshold (and might pass it again soon).

Finally, all active tasks are counted for each queue when determining if a queue is in “danger”. Thus, there may be times when the indicator will light up but you’ll be unable to do anything about it—for example, if you’ve already done all the reviews you’re allowed to do for the current day in that queue, or if all the items to be reviewed are your own work.

Answer 1

Don't show the indicator when that user can't do anything about it.

The indicator is intended to draw attention to the review icon, but showing it when a user can't do anything trains the user to ignore the indicator and the review icon. That's the exact opposite of what's desired from showing such an indicator. The indicator should never be shown when it's not possible for that user to actually review posts in the queue(s) which are "in danger".

While doing this might be more expensive with respect to database access, it's important if we are not going to train users to ignore this notification.

Answer 2

Thanks so much for sharing this information! I'm glad to have it so that we know what to expect. My understanding is that these are (for now) pretty rough numbers so my concern (particularly with the smallest number for the close vote queue being 50) is that many of these queues will never light up red.

I use (nearly exclusively) what I think are generally considered "small" sites... they're the sites that fall into the "default" range on this list but none of them ever have had 50 questions in the close queue that I've seen.

Using your query on the sites I do use, I got the following numbers for the close queue:

• Cooking: 4
• Movies & TV: 15
• Interpersonal Skills: 3
• English Language Learners: 10
• Arts & Crafts: 0

That's a lot lower than 50. And, I'll be honest... I consider M&TV to have pretty high volumes of questions in their close queue and I still wasn't expecting it to be set quite so high.

So, the question is - is it better to start customizing these on a per-site basis to reflect their actual usage or is it better to add additional levels so that all sites are on some sort of level with other sites but there are more than four of them or do we add something additional, particularly for really tiny sites, that will help them find their way to the queues?

My concern with per-site customization is that it's difficult to document - your chart in the question is clear and easy for everyone to understand. If each site gets to set their own limits, this becomes tedious to document. At some point we may decide that it's necessary but, until then, I think we should try to make it simple.

My proposal:

Add a fifth group... call it "Small", maybe? This is set much lower than the others - 10 for close votes and 3 for everything else. This means that ELL and M&TV will turn on occasionally but the others won't... which is OK, because there's another thing you add that will only affect these little sites even though you can implement it across the network.

Set a timer.

If something has sat in the review queue for more than X days... say one or two... that red dot lights up even if there's only one thing in the queue because clearly, no one is looking for it and we don't want to let these age away on our tiny sites just because they're rare and users don't notice the queues.

---

There are certainly other options I've been considering but I think that they'd likely be more complex to implement. This proposal only requires a fifth group and a timer.

My favorite/preferred enhancement to the idea above would be:

Light the indicator when the total number of reviews meets N₁, in addition to when a specific queue does. This would particularly help small sites where they may have one or two items in each queue but never get to that magic number for all of them. So, say N₁ is 12 for small sites, if a site gets:

(That's - 7 close votes, 2 first posts, 2 low quality posts, 1 reopen vote)

The red dot lights up in the top bar (though the queues stay grey) despite none of them meeting the red dot limits. I recommend this combined N₁ should be something just slightly higher than the close vote queue, scaled e.g.: 10250, 230, 115, 55, 12.

This would help sites that are really close to the upper end of their thresholds but don't quite get there regularly. Coupled with the timer function mentioned earlier, this should help bring more visibility to the queues without (hopefully) sending out the bat signal constantly and annoying people.

Answer 3

What I am missing so far is an indicator for when there are any items to review.

At the moment the only time the overall review indicator on the icon lights up is when there is at least one red dot in a queue, but as Catija already explained in her answer this is rarely ever the case on small sites. And even if none of the queues are in utmost danger, I'd still like to know when there's something to review (i.e. a queue has a grey dot). This was a very significant feature from the previous system that has gone lost and I can't really imagine people to review more now that there isn't any kind of alarm for it anymore.

• The easiest and most straight-forward option would be to just light up a grey dot on the review icon when there are items in any of the queues (i.e. any queue has a grey dot).
• Maybe that's considered too much and would be on most of the times, but this can be mitigated by a timer similar to the one already done for the red dot.
• Alternatively the threshold could be a little higher than one queue being grey. But the point I'm trying to make is that this threshold should be significantly smaller than the current red-only one.
• If grey is considered inappropriate, you could as well pick something else, like yellow, or maybe even a multi-layered system with yellow-orange-red (green might be confusing). But something as elaborate isn't even necessary, on the bottom line

just show us when there's anything to review.

Answer 4

Isn't the color of the grey dots too light? Sometimes, I barely can see them.

Current color:

It would be better something a little bit less lighter: