10

There's been a lot of discussion about the review queue in the top bar. Yes, a lot of it focuses on the icon - that's not what this is. I want to talk about getting eyeballs in review and doing that is all about when the icon's indicator is illuminated and when it is not.

Is it broken?

There are a couple of related feature requests on the "New top bar is live" post.

Both of these have a reasonable number of upvotes, so clearly we want some functionality out of this indicator that we're not getting, so I'd say that, yes, there's room for improvement. I'm pulling it out of the top bar post because I think it's worth discussing.


How it works:

Right now the red indicator only illuminates when a specific situation is met (based on this post by Jon Ericson):

  1. if any review queue has a red dot (explained below) and
  2. if it’s been at least an hour since you clicked on the review icon.

Each queue may be in one of three states:

  1. No dot (empty queue)
  2. Grey dot (at least one but fewer than N review tasks)
  3. Red dot (N or more review tasks)

Each review queue has its own threshold (N). [see linked post for numbers]

There is no indicator at in the top bar (only in the drop down) for any number of grey dot review items. Grey dots appear with as few as a single review item up to the quantity that causes the red dot.

My concern is that this is causing fewer users to go to the review queues unless there is a greater-than-average number of items to review because there is no obvious signal that there is anything to do without clicking on the top bar icon to check. Some data on this would be interesting.


Potential talking points

Here are some of the options I've thought of in relation to this. You can ignore or use them as you like.

  • Should a grey dot (or some number of grey dots - 2 or 3) illuminate the indicator?
    • If so, should it be grey or red? If grey, should some number of grey dots escalate this from grey to red?
    • Should the grey dot indicator also rest for some amount of time? An hour? Two?
  • How should the grey/red threshold be determined?
    • Should the dot be illuminated based on total review items rather than per-queue review count?
    • Should this continue to be on a per-site, average queue size basis? Change it to three levels where any site is either level 1, 2, or 3? What would that entail?
    • How can we let users know what this threshold is? Should we even?
    • Should we be concerned about this (potentially) being different for every site?

So, what solutions are there to this? How can we encourage users to check the queues without (especially on SO) causing dot fatigue? How can we make this simple to explain and track, like reputation levels?

  • 3
    You do realise now someone has to make a userscript to turn them all into batsignals... or the troggy signal meta.stackexchange.com/questions/121570/… – Journeyman Geek Oct 17 '17 at 0:16
  • 1
    Also, I like the idea of multiple icons reflecting the need for someone to look at queues over the current "its fine" "some items" "PANIC" system. Not sure how visible they would be though. – Journeyman Geek Oct 17 '17 at 2:40
  • 3
    Keep in mind that there's a real danger in getting too many people looking at review when there are only a few items available: some of them won't get to review anything. One of the most persistent complaints with the old system was caused by items in the queue being either previously reviewed (and so unavailable to the user looking to review them) or assigned to others (and thus similarly unavailable). This can and does change while you're in the process of moving your mouse to the queue in the list, so it's already reasonably likely with a threshold as low as 3... – Shog9 Oct 17 '17 at 3:41
  • @Shog9 I'm sure that's a concern on some sites but I don't really think it's always the case. On M&TV, for example, there are 7 items in the close vote queue that I'm guessing have been there for a few hours... I don't have a way to tell but it was 8 items when I wrote this post, and I'm guessing they haven't gotten a completely new set of them in the interim. An alternate option could be basing it on the age of the oldest review item...? Then if something's sat in review for a day and no one has seen it, clearly it needs more attention. – Catija Oct 17 '17 at 3:44
  • There are lots of ways we could do it, but practically-speaking the easiest fix is to just lower the threshold for that queue on M&TV, @Catija. I pulled the current thresholds from about three months worth of data across all sites, grouped into roughly three categories (small, medium, Stack Overflow) - that means right now M&TV has the same thresholds as Gardening and Database Administrators, sites which have a 10x difference in questions per day. There's nothing magical about three strata, I just needed to start somewhere... – Shog9 Oct 17 '17 at 3:50
  • @Shog9 I was hoping that would be part of what this post was for... figuring out whether there's a better way to do it that helps more people actually get to the queues. I've seen a lot of people frustrated that they're unaware that there's anything in the queues at all because there's no indicator for "a few" things, only for "the queues are on fire, go put it out". Yes, clicking on the talking stapler doesn't take long but you also have to think about it. If there's a dot there, even if it's grey, you can say to yourself, ah, I see there are some reviews, I'll look at those in a bit. – Catija Oct 17 '17 at 3:56
  • @Shog9 Having a more complete understanding of how they do work would be valuable information, too... and why you chose what you did, what the thresholds are, etc... I've said it before... I love data. I'm sure many of us would like knowing where the current methodology came from - maybe we'll even decide that it's fine the way it is once we understand it... or we could realize that it'd be better to have 4 or 5 groups... or not use volume at all... that's sort of the fun of this sort of thing. – Catija Oct 17 '17 at 4:11
  • 2
    On a very small site such as Latin the queues will never get red. Review items will start expiring from the other end before a critical level is reached. At that site it would make sense to indicate whenever there is at least one item available for review. If the threshold is much higher (like three), most users will never see any signal. – Joonas Ilmavirta Oct 17 '17 at 7:02
  • 3
    Yeah, Jon's been working on a thorough document for that in my absence, @Catija - should be ready to go out soon. Also, the bug you linked to? Turns out it is very much a bug: bat signal is non-operational for folks without the mod-tools privilege. Should be fixed in the next day [21:11 UTC: now fixed]. – Shog9 Oct 17 '17 at 23:49
  • @Shog any stats on the impact of the new top bar on review actions across the network? – Shadow Wizard is Ear For You Oct 18 '17 at 6:48
  • 1
    Too early to tell, @ShadowWizard. Nothing way out of wack thus far. – Shog9 Oct 18 '17 at 21:10
3

I've not been much involved in the way the review indicator was designed. (I was busy with something else.) In addition, the review queues are not something I'm excited about using; that sort of task just doesn't motivate me. I also preferred contributing to very small sites that don't have many reviews. But I do think the review system has been critical in spreading the load of site sanity and I'm happy to jump in to help out if needed. Naturally, I'd design a system for people like me. ;-)

My assumption is that there are some people who occasionally click through to the review queues because they enjoy this sort of cleanup task. I appreciate having these folks around because they keep the queues moving during normal site operation. In an ideal world, there'd be enough regular reviewers to keep the site in shipshape order without calling in other users. However, that's not likely to be the case.

Therefore, I'd like the reviews-needed indicator to work a lot like the bat signal. Most of the time people who enjoy fighting crime handling reviews do the work, but when there's an emergency the signal summons me to action. Ideally, these emergencies would be fairly rare. Maybe once a week? As long as it feels special to get the indicator, I'd be motivated to do a review task or two to help out.

Right now, the bat signal is almost always on at Stack Overflow except when I click to silence it for a half hour. Hitting the snooze isn't very heroic. I'm hoping that now a bug has been fixed that stopped illuminating it for users with less than 10k, the light will stay off. But it's also possible the problem is that there are 8 different queues that might be in the dangerzone at any particular time. It might also be that I'm in an unfortunate timezone and that things would be better if I were further east.

On smaller sites, it seems not to be lit up much at all. Many sites are hovering right around the threshold for one or more queue (often close votes). It might be that the signal is working just the way I'd like it. But I can think of a few pathologies:

  1. Review tasks aren't getting done, but just sitting unhandled for days on end. On the rare occasion the indicator lights up, someone handles a few reviews to turn it off. Then the queues go quiet as nobody bothers to do reviews without the dot. Maybe close votes are expiring because nobody looks at the queue. Cruft starts to accumulate on the site.

  2. People who would like to do reviews don't know that there are tasks available because they assume there are no tasks unless the the red dot appears or don't think to try the icon. They might be enjoy the site better with more reviews in their life.

  3. Since the signal is hardly ever on, users get really concerned when it does light up. As a result, review queues are cleared in fits and starts. Also, there's an unsettled feeling from folks panicking unnecessarily.

It might be that we could adjust the thresholds to where the indicator comes on regularly yet rarely. That would probably solve the problems I can see on small sites. If not, I do like your timer idea. My suggestion would be to turn on a grey indicator if there hasn't been a review task accomplished in X hours. (X should probably not be 12 or 24 to avoid timezone bias.) Obviously Stack Overflow and other large sites would never get this, but on small sites the occasional grey indicator would remind people to do a review once in awhile without the implied danger of a red indicator.

|improve this answer|||||

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .