Experiment: Review-needed indicator logic for sites that sometimes have empty queues

Question

One of the more common bits of feedback when we changed the review queue indicator was that some sites would prefer to have an indication when there are any outstanding reviews available. So we are trying an experiment with an "inbox zero" review indicator. This would mean:

1. The review indicator lights up whenever there is at least one item in any queue.

2. There's no concept of the "danger threshold"; if there's even one item in a queue, it's marked with a red dot.

Clearly, this logic would be annoying for Stack Overflow's Close queue, so we've made it a site setting. We're going to test it out on a handful of sites to see how people respond to it:

• Meta Stack Exchange
• Movies & TV
• Internet of Things
• Martial Arts
• Role-Playing Games
• Worldbuilding
• Arts & Crafts
• Unix & Linux
• Computer Science Educators

One final thing: you will be notified even when you cannot complete a review task (for anyone who can see the review queue - 500+ reputation). In particular, people who do not have enough reputation to handle items all the queues will be notified of items in queues they can't access. This is the next thing we are working to fix. If too many people are frustrated by this situation, we'll pause the test until that problem is fixed. It'll also light up even after you've finished all the review tasks available to you, which is a much harder problem for us to fix. The goal of this experiment is to test whether aggressively notifying users can make "inbox zero" a possibility for sites that are close.

Answer 1

One final thing: you will be notified even when you cannot complete a review task. In particular, people who do not have enough reputation to handle items all the queues will be notified of items in queues they can't access.

Please don't do this. You know when you go to some random website that you don't even have an account on and it's got a fake bar at the top with a "notification" lit up to make you click on it, and it doesn't really go away or do anything?

I know you don't mean it to be, but this is like that — but for logged-in users, even! Over time, it will simply teach people that those notifications are frippery and should be ignored.

Answer 2

Early results

In my opinion, the top bar indicator works best when it:

1. Minimizes the time it takes for review tasks to be handled and
2. Minimizes the cognitive impact (to borrow a phrase from this answer).

Unix was the site with the most reviews in the experiment. It's clear that tasks did get done more quickly:

epoch  reviews max minutes pending avg minutes pending
------ ------- ------------------- -------------------
before    2165                7931                 168
after     2553                1690                  56

I won't go through all the sites, but each one saw similarly positive decreases in review time. But meta feedback noted:

As in inveterate queue clearer, the behavioural impact is negligible. The cognitive impact, however, is unwelcome. The red dot immediately signals dread: "Oh, there are a heap of these things to deal with now..."

Save the red for when you really need it (ie., when the queue really is backed up: 10 or more, or whatever the old threshold was). Bring back the grey so the site feels less like it is on fire all the time.

Meanwhile on RPG:

I'm appreciative of the change: if I see no indicator, I now know there's no reviews.

It used to be there could be 1-2 sitting there and I'd just have nothing telling me.

It's a bit over-tuned in that it shows me everything I can't even work on, as mentioned in the question, but over-tuned is a nice departure from under-tuned, and I appreciate there's further work to tune it just the right way.

Reviews on that site also got a lot faster:

epoch  reviews max minutes pending avg minutes pending 
------ ------- ------------------- ------------------- 
before     465                1058                 112
after      437                1012                  58

But notice there are a lot fewer reviews. In the last 9 days, they've had about 2 review tasks per hour compared to Unix's ~15. (If you are doing the math with me, note that RPG started the experiment on the 13^th and Unix started on the 15^th.) Presumably Unix has more reviewers too, but since the average review hangs around for an hour, it's quite likely the indicator will come back each time the hour-long grace period expires after clicking on it. That's clearly too much and a long way from our (somewhat arbitrary) warn 10% of the time criteria.

So I think it's probably reasonable to turn on the new logic for sites with only one or two reviews in an hour. It's not at all reasonable for sites that get double digit review tasks an hour.

False negatives

Several people have noticed that sometimes the indicator doesn't light up even when there are tasks in one or more queue. After you click on the review queue icon, you get an hour-long grace period from being notified about new items. But the reports generally take that behavior into account. It looks like there's another reason for the occasional false negative: caching. So there's a potential delay of a few minutes between when a task enters review and when the indicator might first light up. When someone clicks the review icon, the number of pending tasks for each queue is recalculated and the grace period starts for that user.

Given that reviews are getting handled more quickly, this doesn't seem like a critical problem. The indicator does come on eventually, so it avoids the main criticism of the danger threshold version on small sites. Obviously, it would be nice to have a consistent mental model. On the other hand, it does reduce the potential for overwarning.

False positives

We did get one report of a false positive. In this particular case, it seems the user addressed all the review tasks but they didn't get cleared from the queue within the hour. We are addressing another potential problem: being warned of review tasks when you don't have enough privileges to see the queue. (The work has been completed, but it's probably not a good idea to deploy just before vacations.) False positives are certainly discouraging, so we'll want to eliminate as many as possible, if not all.

Answer 3

_{copied over from https://unix.meta.stackexchange.com/a/4724/246819}

I definitely agree the review queue indicator needs to be more aggressive, but it's neither a good idea nor necessary to sacrifice the urgency indication feature just for that. Instead of setting the whole site on fire, how about a yellow-to-red transition?

One could even think about always having a dot on every queue, maybe green for empty ones – those traffic light colors are quite unambiguous.¹

Of course the color of the review button dot should always match the darkest yellow/red, and there should be no dot only if every queue is green.

1: “Red” meaning „Stop! Don't add any questions to this queue before you did your review duty!“ ;-)

Answer 4

We're taking part in the experiment with IoT.SE, but I'm not sure it's working as intended. I would have expected an overall indicator up there. It was directly after I loaded the page and I clicked more out of habit on the review button.

Answer 5

feature-request

Can the Suggested Edits be excluded from the one-hour-sleep? That queue always had some sort of special handling in the past, even in the old system. Now it doesn't. Today I clicked—by total chance or habit—on the review item while it was in sleep mode with the soothing gray check mark, and there was a suggested edit pending.

The indicator should always light up if a suggested edit is available.

Answer 6

This will not have much impact on me so long as it doesn't notify me about reviews on other sites. The problem is less that I'm doing something else on Politics.SE and don't see that there's items in the queue. It's more that I'm currently on Skeptics.SE and don't have a single step way of checking whether there's action on Politics.SE.

If you really want me to process more reviews, consider offering some sort of combined queue notifier. Notify me if there are any queues that I could process that have open items anywhere. I occasionally review queues on seven sites: the five that appear at the top of my sites menu; Arqade (my normal entry point to the site); and more rarely Economics.SE.

As is, I can't find out that a site needs attention until I get around to visiting it. It's not so difficult to check queues at the beginning and end of my visit to a site, which catches everything that this indicator would. It is difficult to know that just after I left, Politics.SE had a high priority review develop.

I don't know how common my usage pattern is, but I don't believe that I'm unique. I see others on multiple sites and see people talking about their activity on other sites.