105

A few weeks ago, we tested a modified review queue indicator for small sites. In essence, the top bar icon will get a red dot whenever there's any task and not just when the count reaches a discrete threshold. The experiment seems to have worked for small sites, but can be annoying when there are many active reviews. I'd like to come up with an objective criterion we can apply to determine which version to deploy to each site. (And, as sites grow, there will be a criterion to know when we ought to change the setting to the danger-lever version.)

I should note that:

  1. People are no longer notified of review queue tasks that they can't access because of their privilege level and

  2. People might still be notified of review tasks they have already completed if they don't get cleared after an hour

As a result, false positives are still possible for sites with few non-moderator reviewers but should be rare otherwise.

Proposed criterion: maximum median tasks per hour <= 2

The original design for the indicator was built to fire about 10% of the time. The notify-immediately version of the indicator will alert more often than that unless a site is extremely slow. But that's balanced by the benefit of being able to clear the review queues more often. If the typical hour on a site has only a few items to review, you might not mind getting notified every hour since you can clean them out each time. But if the typical hour has many more items than anyone can reasonably review, the alert is of the worst kind: constant and uninformative. By this criterion, here are the statistics for the experiment sites for the first two weeks of December 2017:

Site                       Queue           Median hourly tasks 
----                       --------------  ------------------- 
Meta Stack Exchange        Close Votes                       1
Movies & TV                Close Votes                       5
Internet of Things         Close Votes                       2
Martial Arts               Suggested Edit                    0
Role-Playing Games         Suggested Edit                    0
Worldbuilding              Close Votes                       2
Arts & Crafts              Suggested Edit                    1
Unix & Linux               Close Votes                       9
Computer Science Educators Close Votes                       1

Unix clearly has too many for the indicator to be helpful. (I ended the experiment for them last week.) Yes, they normally have few tasks outstanding because the community is very responsive. Warning sooner did reduce the time review tasks had to wait, but the satisfaction of completing the queue is ephemeral as new items arrive constantly. Meanwhile, there are hours in which nothing new comes to even RPG's most active queue. (The chart suggests it is Suggested Edits, but since the median is 0, it could be any of the other queues are more active. Depends on how SQL sorted the results.) So I think the median tasks for the most active queue is a pretty good metric.

As for the actual criterion, 2 seems reasonable to start with since:

  1. It mostly divides the experiment sites between between those that appreciated the change and those that didn't and

  2. It's easy to imagine someone doing a couple of tasks between answering questions and not feeling put out by the interruption

The caveats for those points are:

  1. Movies & TV seems to like the new indicator and would fail this criterion and

  2. This assumes that the other queues are basically empty and that reviews are not time consuming

Rollout process

I began rolling it out on sites with zero median tasks per hour. Assuming all goes well, we'll move on to the sites with a maximum of one per hour:

  • Fitness and Nutrition
  • Motor Vehicle Maintenance and Repair
  • Parenting
  • Aviation
  • Poker
  • Space Exploration
  • Reverse Engineering
  • Language Learning
  • Hardware Recommendations
  • Literature
  • Politics
  • ExpressionEngine
  • Mythology
  • Genealogy and Family History
  • Arts & Crafts
  • Embedded Systems
  • Puzzling
  • Vegetarianism
  • Joomla
  • Interpersonal Skills
  • Mathematics Educators

What am I missing and what should I look for when the change goes to more sites?

31
  • 26
    With 170 sites, it's not practical to ask each meta to give us their preference. Huh? How is that a problem? 170 isn't actually that much. It'd be one single meta per site, and in the end, you'll hear one voice per site anyway.
    – Möoz
    Jan 18, 2018 at 21:41
  • 6
    @Möoz assuming there's not a way to automatically post the question on each meta, that would take hours of work to complete, even if only the lower-volume sites were addressed... and that's just to post the questions, not to respond to any requests for clarification. The easier (for the CM team) option would be for the CMs to mobilize the mods to do this for them and have sites that opt-in contact the CMs... but what they've chosen to do is even easier... and, presumably sites that don't like it can complain and have it removed... or sites that want it can ask for it as one-offs.
    – Catija
    Jan 18, 2018 at 22:57
  • 15
    @Catija that would take hours of work to complete You're right, the CMs should do less of their work.
    – Möoz
    Jan 18, 2018 at 23:52
  • 14
    @Möoz I'm not quite sure what you mean by that? There's a concept of working smart ... the feeling I've had for the past year or more is that the CMs seem pretty overworked and don't have enough time to do the things that need to be done... let alone reaching out to each site when some data like this can make it less necessary... and (I think) what a site wants in this situation without having tried it may not be of much use... often people prefer the status quo over something new and different, even if it's actually better.
    – Catija
    Jan 19, 2018 at 0:09
  • 21
    "We're satisfied with the way the actual indicator works for now": You might be satisfied, but since you changed the indicator, I've stopped accessing the review queues because that looks like a dead menu. Am I the only one thinking this? It's beyond my understanding why you don't simply indicate the number of review tasks one is expected to do. Jan 19, 2018 at 19:15
  • 5
    @MassimoOrtolano: Yes that number was an ideal solution. The trouble is the number was often misleading due to heavy caching and necessary assumptions to keep this feature from killing our SQL server. I'm gathering the data to write up a summary of the change and it does seem there was an increase in reviewers immediately after the change (which is to be expected) followed by a drop off in December (also expected). So far in January, the number of active reviewers has returned to about the level it was with the number instead of the indicator. I'll link to more details after I post them. Jan 19, 2018 at 19:35
  • 7
    @Möoz: I've been thinking about your comment, the nature of my job and the various ways we involve communities in changes to the underlying system. The easy thing would be to just make the change on the sites I think could use it. But I see this as an opportunity to involve users in determining how these sites work. Writing this question clarified my thoughts and helped me pick a better criteria than the one I started with. If you'd like to suggest a better criteria (including asking on each meta), I'm open to hearing it. That's why I bothered to ask in the first place. Jan 19, 2018 at 20:47
  • 4
    Will this also come to beta sites? This change would be very welcome on TGO (at least by me), as I never see any status indication at present (I just click the review menu sometimes when accessing the site).
    – imsodin
    Jan 19, 2018 at 21:50
  • 6
    @imsodin: Yes! The Great Outdoors has a median hourly review queue size of 1, so it'll be coming in the next wave. Jan 19, 2018 at 21:56
  • 3
    FWIW I think that the metric is fine for less "operation critical" queues like Low-Quality or even suggested edits. I'd want to be notified ASAP about any Close Queue Task I can do on the sites that I review on, though. Close Votes are one of the easiest barriers to low-quality answers, given they're applied in a timely manner. Such timely manner is only achievable with a proper notification in each case. The solution is to fix the caching for smaller sites... Jan 21, 2018 at 20:59
  • 10
    @JonEricson It's not that you prefer to do things the easy way (I understand you guys are busyAF), it's that you blatantly tell us that contacting us is something you have to be bothered to do and you see it as effort. Honestly, in this case, it would have been better to just roll it out and go with what you were thinking, rather than say that. Writing this question clarified my thoughts and helped me pick a better criteria than the one I started with might be the only good thing that came out of this. Your attempt at 'involving' us seems to be half-hearted at best.
    – Möoz
    Jan 21, 2018 at 22:00
  • 10
    @JonEricson Now don't get me wrong, I seriously appreciate the hard work you guys do. It just hurt to hear that I may be involved in one of the 170 sites that are difficult and a 'bother' to contact.
    – Möoz
    Jan 21, 2018 at 22:02
  • 8
    @Möoz: Ah. Yes, that was insensitively worded. Truth is, we typically roll out changes and there's no space for community feedback. For this particular change, I had a chance to involve sites. The initial contact on meta isn't impossible, but there's the a good chance the feedback would create ongoing work that probably isn't necessary if there's an objective criteria. So that's what I was getting at. I'm sorry I caused (or increased) pain for you. :-( Jan 21, 2018 at 23:04
  • 6
    I apologize that I don't have time to read much of what is written here right now. My experience is that I regularly see review dots (especially red ones), yet there is nothing to review. Sometimes there is; sometimes there is not. I've learned to ignore the dots because they seem to not accurately indicate if reviews are waiting in the queues. That's just my experience, but I thought sharing it might be helpful. Jan 23, 2018 at 18:29
  • 16
    You should stop bike shedding minor aspects of the review queues and work more on 1) making the review process friendlier (like by adding downvoting and other actions into it, making audits not irritating), 2) making the review process more effective (so participants actually make a dent in all the crap on the sites), and 3) blocking stuff automatically so it doesn't have to get into the review queue in the first place. (I might try to work this into a full answer when I have time.)
    – jpmc26
    Jan 23, 2018 at 18:54

5 Answers 5

71

I think that this statement:

people might still be notified of review tasks they have already completed if they don't get cleared after an hour.

is a bigger issue than you are allowing for.

Why do we care about false positives?

On a particular small site that I am an active reviewer on, the close vote queue can sit with active items for entirely too long. Given the number of votes it takes to close items, this is not surprising in a smaller community. One solution for this problem, is for the moderator to come through and impart their will for the entire community, by processing the queue. While this will clear the queue, it will not allow the site to build a community of reviewers as the site tries to move from small to (hopefully) larger.

Therefore, the close vote queue will routinely not be cleared in an hour. So, what is the result? The indicator is then frequently on for those who have gone through the queue recently (last day or two). So, those who might actually be willing to go to the queue when needed, will get trained to ignore the indicator because it is too often wrong.

I realize that SO regulars will laugh at the notion that the close vote queue could possibly be cleared in a timely manner, but... it is possible. Unix, where I have been an active reviewer, and who was featured in your examples, manages to get their close votes processed on a regular basis.

Answer to your question:

  • Yes, Please

    On sites with few reviews per hour, notify about anything to do in the review queues. Less than 2 per hour seems like a reasonable threshold.

  • No, Thank you

    Please don't lie about there being something to do in the queues.

10
  • 43
    I fully agree. On a small site (like Latin) it can take days to accumulate five votes to close a post but new closable items don't enter the queue every day. If there are false positives, they easily become the rule rather than the exception, disincentivizing the few active reviewers. Jan 20, 2018 at 11:03
  • 6
    Fully this. Was the reviewed indicator tested for large and small sites separately? If not, I'd suggest an additional A/B test there, to see how bad things would become.
    – FooBar
    Jan 20, 2018 at 15:26
  • 18
    To put it more bluntly: I will quickly start ignoring the indicator if it makes me go to the review page for nothing. The trouble with these things is: trust leaves by horseback and returns by foot.
    – Jan Doggen
    Jan 22, 2018 at 9:20
  • 2
    They've been ignoring this issue for years - it really needs to be a higher priority to fix! Jan 22, 2018 at 12:21
  • I'm thinking the best next step is to enable the new indicator on the first wave of sites and see how often we have this problem. One solution for very small sites would be to extend the grace period for people seeing the indicator after they dismiss it. Jan 22, 2018 at 17:31
  • 7
    Yes. How about green for 'clean' (and thanks), an orange dot for 'something to do' and a red dot for 'do something!, please'. We don't want stuff to sit unedited while responses are made to it, nor do we want 'cry wolf queues'. Sometimes stuff simply languishes on busy sites because the people who really like to edit and maintain order (cough, photo) happen to be offline and sometimes there's a cleaning spree where all the posts are 'attacked' and the whole 'new question queue' is all 'modified by user #$'. There must be a single adaptive algorithm that can simply be deployed everywhere.
    – Rob
    Jan 23, 2018 at 3:21
  • 8
    @JonEricson On MA (and others) the problem has caused me (at least) to ignore the silly red dot as it often leads to nothing more than a waste of time! Jan 23, 2018 at 9:33
  • 4
    Came here to complain about a false positive on the review queues on a child site. I don't care about the size of the queue, especially for smaller sites. I just want an accurate notification that there is something I can review in a queue, so I can review it. Jan 23, 2018 at 14:23
  • 2
    @JonEricson It's hard to understate the importance of Sardathrion's point there. Might it be helpful to provide some appropriate bibliography as references?
    – E.P.
    Jan 25, 2018 at 16:06
  • @Gray, thanks a lot. :-) . 1200 character limit, something had a to go. I will take another look for something more reasonable. Mar 19, 2018 at 20:35
8

I just changed the setting on the following sites:

  • Science Fiction & Fantasy
  • Programming Puzzles & Code Golf
  • Webmasters
  • Retrocomputing
  • Role-playing Games
  • Board & Card Games
  • Vi & Vim
  • Community Building
  • Sitecore
  • SharePoint
  • Italian Language
  • Sustainable Living
  • Anime & Manga
  • Robotics
  • Spanish Language & Usage
  • Martial Arts

Feb. 15 update. I've expanded the list to the following sites:

  • Fitness and Nutrition
  • Motor Vehicle Maintenance and Repair
  • Parenting
  • Aviation
  • Poker
  • Space Exploration
  • Reverse Engineering
  • Language Learning
  • Hardware Recommendations
  • Literature
  • Politics
  • ExpressionEngine
  • Mythology
  • Genealogy and Family History
  • Arts & Crafts
  • Embedded Systems
  • Puzzling
  • Vegetarianism
  • Joomla
  • Interpersonal Skills
  • Mathematics Educators

April 20, 2018: Added The Great Outdoors.

If you notice any problems, please post here on meta.

Assuming all goes well, we can do the next two tiers in the coming weeks.

18
  • I'm a moderator and Webmasters and I'll keep my eye on it. I would certainly like to be notified of anything to do in the queues there because there is usually nothing to do. Jan 29, 2018 at 20:05
  • When I clicked on the queues icon on Webmasters I found a red icon next to the close votes. There had not been a red circle on the top bar. I might be misunderstanding this, but it sounds like there should have been an indicator on my top bar. Either that or clicking on the queues icon gets fresher data than when I loaded the page. Jan 29, 2018 at 20:18
  • @StephenOstermiller How often do you check the review queues? The red indicator will only light up once per hour and checking it resets it to "off", so if you'd checked it within the last hour, it won't light up... if that makes sense?
    – Catija
    Jan 29, 2018 at 20:21
  • I rarely check, and hadn't in the last hour. Jan 29, 2018 at 20:28
  • @StephenOstermiller: I'm not entirely sure how it works in the hour or so after I turned it on. If you see the problem again, could you try to capture as much detail as possible and post a question on meta? Jan 29, 2018 at 20:31
  • Will do, I'll keep my eye on it Jan 29, 2018 at 21:08
  • 1
  • "When I clicked on the queues icon on Webmasters I found a red icon next to the close votes. There had not been a red circle on the top bar." I've noticed the exact same behavior in Spanish.SE. No dot in the top bar icon. Click icon out of curiosity. BAM! Big red dot next to the Close Votes link.
    – walen
    Jan 31, 2018 at 16:53
  • Now the behavior is the opposite: the top icon shows a red dot, but there are no dots next to any of the queues when I click to expand the menu: imgur.com/a/UylC5 (and the queues show 0 items if I click any of them).This has happened twice just this morning. I'm not complaining, just giving feedback. If it's not needed, no problem, just say so :D
    – walen
    Feb 5, 2018 at 11:46
  • @walen: It looks like you are running into the problem I mentioned in the question of an outstanding review that you've handed, but has not yet been completed by someone else in the community. The culprit this time is this Low Quality review. I'm glad it didn't bother you too much, but I wish we could fix it. Maybe you could drop a note in the site chat room asking folks to look at the queues next time it comes up? Feb 5, 2018 at 19:59
  • Hey Jon! Does Cooking not apply here? I'm in the close review queue there right now and I've gone through 5-6 items that have probably been sitting around for a while and every single queue has a grey dot.
    – Catija
    Mar 23, 2018 at 21:25
  • @JonEricson On the question you responded that The Great Outdoors would be in the second wave of sites for this change. Now we aren't in there. What changed? There hardly was a significant increase in questions/review items.
    – imsodin
    Apr 19, 2018 at 20:21
  • @imsodin: I'm not sure how that slipped through, but I've updated TGO now too. Apr 20, 2018 at 23:34
  • @Catija: I missed your comment, but Cooking got a median of 17 close vote tasks per hour in the last two weeks. Apr 20, 2018 at 23:36
  • That seems really high... I mean... it's 16 right now... and there's really not that many active close voters, particularly since Stephie became mod... so it might be worth looking at the length of time those items have been in the queue ... I'm guessing they're the same 16 things for a week or more at times... but 16 is a lot to not have a red dot.
    – Catija
    Apr 20, 2018 at 23:43
6

Suggestion: Can per-site metas (with the possible exception of Meta Stack Overflow) get the quick indicator (i.e., red as soon as there is a review task)?

Rationale: Review tasks on per-site metas are very rare and get little visibility (thus taking forever). A quick review indicator could help.

4

Can't we make this a per site user setting?

Is there any reason not to? It could be just a checkbox on the review queue overview page.

☐ Show review queue indicator for this site.

2
  • 8
    First, that's not a very accurate way to describe the setting. Second, this is an example of a harmful user preference. Generally, it's better to fix the default setting so that everyone can benefit. Jan 22, 2018 at 17:31
  • 6
    @JonEricson in that case, turn them on, everywhere, all the time.
    – SQB
    Jan 22, 2018 at 19:07
3

I would like to have a traffic light indicator (or something alike to consider color vision impairment) for each queue to see what is more urgent, and one general indicator for the site. This is similar to the proposed change by dessert.

I don't know how if the time frame is correct, maybe it needs to be (automatically) adjusted for each site by considering the number of possible reviewers and the actual review work and the number of reviewers.

Thresholds for each queue:

  • grey: any action available
  • orange: a quarter of (new?) actions (of the previous week?) accumulated
  • red: half (new?) actions (of the previous week?) accumulated

General light for all the queues:

  • orange/grey: one quarter or more of the queues (available to the user) are in orange
  • red: half or more of the queues (available to the user) are in orange

I think this would help to send a signal to users who don't go often to the review queue, but would find some queues to work in. It is not fixed to a certain amount, so for sites that work well the general indicator wouldn't bother users not interested in doing review, unless the usual reviewers can't handle the workload. Is quite similar to how it actually works (so maybe it isn't worth much or should the timing frame be one hour instead of quarters of the previous week).

This will probably mitigate the effect of new reviewers fading away after a change on the notification system because the notifications will depend on the community work itself.

1
  • 3
    I like basing the indicator based on average queue size from the last week. I think that would work on both big and small sites. If the queue is some percent higher than it has been, fire the beacons -- Gondor calls for aid! Jan 29, 2018 at 20:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .