Review Queue Bat Signal - How can we make it more useful?

Question

There's been a lot of discussion about the review queue in the top bar. Yes, a lot of it focuses on the icon - that's not what this is. I want to talk about getting eyeballs in review and doing that is all about when the icon's indicator is illuminated and when it is not.

Is it broken?

There are a couple of related feature requests on the "New top bar is live" post.

• This one is requesting that users be notified when there are grey dots in the review queues. +7 as of writing
• This one is more of a bug report saying the same thing. +12 as of writing

Both of these have a reasonable number of upvotes, so clearly we want some functionality out of this indicator that we're not getting, so I'd say that, yes, there's room for improvement. I'm pulling it out of the top bar post because I think it's worth discussing.

---

How it works:

Right now the red indicator only illuminates when a specific situation is met (based on this post by Jon Ericson):

1. if any review queue has a red dot (explained below) and
2. if it’s been at least an hour since you clicked on the review icon.

Each queue may be in one of three states:

1. No dot (empty queue)
2. Grey dot (at least one but fewer than N review tasks)
3. Red dot (N or more review tasks)

Each review queue has its own threshold (N). [see linked post for numbers]

There is no indicator at in the top bar (only in the drop down) for any number of grey dot review items. Grey dots appear with as few as a single review item up to the quantity that causes the red dot.

My concern is that this is causing fewer users to go to the review queues unless there is a greater-than-average number of items to review because there is no obvious signal that there is anything to do without clicking on the top bar icon to check. Some data on this would be interesting.

---

Potential talking points

Here are some of the options I've thought of in relation to this. You can ignore or use them as you like.

• Should a grey dot (or some number of grey dots - 2 or 3) illuminate the indicator?
  • If so, should it be grey or red? If grey, should some number of grey dots escalate this from grey to red?
  • Should the grey dot indicator also rest for some amount of time? An hour? Two?
• How should the grey/red threshold be determined?
  • Should the dot be illuminated based on total review items rather than per-queue review count?
  • Should this continue to be on a per-site, average queue size basis? Change it to three levels where any site is either level 1, 2, or 3? What would that entail?
  • How can we let users know what this threshold is? Should we even?
  • Should we be concerned about this (potentially) being different for every site?

---

So, what solutions are there to this? How can we encourage users to check the queues without (especially on SO) causing dot fatigue? How can we make this simple to explain and track, like reputation levels?

Answer 1

I've not been much involved in the way the review indicator was designed. (I was busy with something else.) In addition, the review queues are not something I'm excited about using; that sort of task just doesn't motivate me. I also preferred contributing to very small sites that don't have many reviews. But I do think the review system has been critical in spreading the load of site sanity and I'm happy to jump in to help out if needed. Naturally, I'd design a system for people like me. ;-)

My assumption is that there are some people who occasionally click through to the review queues because they enjoy this sort of cleanup task. I appreciate having these folks around because they keep the queues moving during normal site operation. In an ideal world, there'd be enough regular reviewers to keep the site in shipshape order without calling in other users. However, that's not likely to be the case.

Therefore, I'd like the reviews-needed indicator to work a lot like the bat signal. Most of the time people who enjoy fighting crime handling reviews do the work, but when there's an emergency the signal summons me to action. Ideally, these emergencies would be fairly rare. Maybe once a week? As long as it feels special to get the indicator, I'd be motivated to do a review task or two to help out.

Right now, the bat signal is almost always on at Stack Overflow except when I click to silence it for a half hour. Hitting the snooze isn't very heroic. I'm hoping that now a bug has been fixed that stopped illuminating it for users with less than 10k, the light will stay off. But it's also possible the problem is that there are 8 different queues that might be in the dangerzone at any particular time. It might also be that I'm in an unfortunate timezone and that things would be better if I were further east.

On smaller sites, it seems not to be lit up much at all. Many sites are hovering right around the threshold for one or more queue (often close votes). It might be that the signal is working just the way I'd like it. But I can think of a few pathologies:

1. Review tasks aren't getting done, but just sitting unhandled for days on end. On the rare occasion the indicator lights up, someone handles a few reviews to turn it off. Then the queues go quiet as nobody bothers to do reviews without the dot. Maybe close votes are expiring because nobody looks at the queue. Cruft starts to accumulate on the site.

2. People who would like to do reviews don't know that there are tasks available because they assume there are no tasks unless the the red dot appears or don't think to try the icon. They might be enjoy the site better with more reviews in their life.

3. Since the signal is hardly ever on, users get really concerned when it does light up. As a result, review queues are cleared in fits and starts. Also, there's an unsettled feeling from folks panicking unnecessarily.

It might be that we could adjust the thresholds to where the indicator comes on regularly yet rarely. That would probably solve the problems I can see on small sites. If not, I do like your timer idea. My suggestion would be to turn on a grey indicator if there hasn't been a review task accomplished in X hours. (X should probably not be 12 or 24 to avoid timezone bias.) Obviously Stack Overflow and other large sites would never get this, but on small sites the occasional grey indicator would remind people to do a review once in awhile without the implied danger of a red indicator.