21

On a number of sites*, there's enough review flow to reliably keep certain queue backlogs in the dozens or even hundreds. This is discouraging to regular reviewers; some sites struggle recruiting more than a few reviewers, and a number of comments have cited large queue lengths as the demotivating factor limiting their own involvement. It also increases time to handle new items, making community moderation less effective and causing more work to clean up e.g. answers to questions that are not closed promptly.

The only feature that currently acts to actively restore the balance in most such cases is the raising of daily caps to 40 instead of 20. But because this takes effect only when the queue has a backlog of 1000 or more, and cuts out as soon the backlog drops below that (without waiting for any user to finish with their expanded allowance), it does nothing to actually bring the queue to 0. Clearly, though, the general approach is considered philosophically acceptable: if there's just too much in the queue, give each willing reviewer somewhat more opportunity to try to take care of it all, even at the expense of slightly reduced accuracy or higher per-reviewer load.

What should be changed, therefore, is the trigger. How about a graduated system? For any week** in which a given queue never reaches 0, the daily cap for all users for the next week** is raised by 10, up to a maximum of 50. Conversely, for any week** in which a queue never reaches 20 items backlogged, the cap is lowered by 10, to a minimum of 10. If neither condition is met, the cap remains the same as the previous week**, and caps are initialized to the result of the current calculations. Transitions can be very efficiently tracked by checking whether a given queue is at 0 or 20+ whenever the 10k-tools review count is updated, and setting a bit field if so.

Some queues are limited by votes, not just the review cap: Close, Reopen, and certain functions of Low-Quality. These should ideally allow additional queue-only votes; specifically, allow 5 review-queue-only close votes for every 10 reviews in the cap (including the baseline). So a queue operating at 50/day/user would allow the first 25 close votes to be cast for "free". (These would probably be shared between all eligible queues, including Triage on SO, since the alternative requires a much more complicated UI.) Delete votes for 20kers in LQP should be handled similarly, but the scaling would need to be rethought; that deserves a separate proposal.

The primary benefit of this proposal is twofold: queues that are not absolutely incorrigible could be brought reliably to heel by the community, removing the futile perception of fighting perpetually multi-hundred-item queues, and newly busy queues would be adjusted for much faster, reducing system lag in most cases of increased load. (Not counting events like the recent Late Answers surge, of course.) Fundamentally, this addresses the key problem of review queues, which is not current backlog, but capacity as a rate at the margin.

The general principle behind this idea has been recently tested at SO, and the results were officially promising enough to warrant some sort of scaling increase. I think this algorithm is one of the best choices for this: it's relatively simple but quite flexible anyway.

*SO's CV queue is of course legendary, and would not be materially affected by this, but the recent growth of LQP there seems to have no real end in sight (900 at time of writing, and still climbing fast). Other sites affected include SF and SU, ELU, Android, DIY, AU, and probably dozens of others I'm less aware of.
**Or day. I'm not sure which granularity is likely to be better.

  • 5
    Thank you for reviewing 20 low quality posts today; come back in 16 hours to continue reviewing.... Yep. – Patrick Hofman Oct 23 '15 at 7:25
  • 3
    I don't understand what will scale? The number of review items you get? If so, that is kind of point less if I have no close votes, flags or votes left, which often is the case now for me. – rene Oct 26 '15 at 19:02
  • @rene: If you run out of flags, I don't know what to tell you. But in LQP, <20kers can keep going until they run out of reviews, in SE anyone can do the same, and in FP and LA anyone with enough flags can go on for just about forever. In the remaining queues, alternating between skipping and using the non-vote review actions (Leave Open, Leave Closed, Edit, etc) on posts that deserve them is still perfectly viable, if a little unintuitive. – Nathan Tuggy Oct 26 '15 at 19:11
  • That will be a lousy experience for me @NathanTuggy but I can see how it might help for less fanatic reviewers. – rene Oct 26 '15 at 19:20
  • 3
    @rene: Keep in mind that this proposal is specifically not intended to do much of anything to help with SO's CV queue in particular, which needs far more help than this. I'm satisfied as long as it a) makes a lot of other queues a lot better and b) doesn't make anything worse. – Nathan Tuggy Oct 26 '15 at 19:22
  • @NathanTuggy on all sites other than SO, questions go to LQP. I am often faced with a question in there that I can't act on because I have no close votes (and so I have to skip). The same applies to first posts if I'm out of regular votes. Sure, I've got 100 flags available... but can't do anything with them. – user213963 Oct 26 '15 at 19:27
  • This won’t solve the problem. See the quotes and stats at the beginning of this answer of mine. – Wrzlprmft Oct 26 '15 at 20:27
  • 1
    @Wrzlprmft: Your answer appears to be geared toward increasing close votes, while this proposal is mostly about increasing queue caps in all queues that need it. There are a number of queues with multiple reviewers capping every day at 20, not 40. That is an unnecessarily low restriction. Citing statistics about total close votes, and how rarely those are capped, is not helpful, because that's a different problem entirely, and the statistics show quite a different story. – Nathan Tuggy Oct 26 '15 at 20:33
  • @Wrzlprmft: I'm not quite sure what your last comment is trying to say. – Nathan Tuggy Oct 26 '15 at 22:13
  • @NathanTuggy: Weird accident, let me try again: I do not see any reason to assume that review queues behave differently from close votes in that matter. – Wrzlprmft Oct 26 '15 at 22:33
  • @Wrzlprmft: When I look at actual review queues I see different statistics. So as far as I can tell, they not only can, but do behave differently. (In particular, LQP and Triage on SO are glaringly obvious cases where nothing even vaguely resembling close vote statistics is in play. Dozens of reviewers are capping every single day.) – Nathan Tuggy Oct 26 '15 at 22:41
  • 1
    @NathanTuggy There is test running on SO (if you didn't notice yet). – Patrick Hofman Mar 17 '16 at 8:00
10

This is an excellent idea, but it has some concerns too that need to be addressed before this can be effective.

  1. Scale close votes, etc. too. In order to be capable of doing anything in the close or reopen queue (and others, just as an example), the close votes need to scale with the increased review limit too. One idea could be to split the on-site close votes from the review votes. This means we don't need to give users 100 close votes, but just their regular amount plus 100 review close votes, the exact amount of the rate limit of reviews on that day. Effectively this means that close votes from the review queue won't count to your day limit any more.

  2. By increasing the number of votes, the effect can be that even more users skip reviewing since others do it already. I like the part of the community that does their review job every single day, but this could put extra load on them instead of others. The question is: how to keep motivating other users to do their review jobs?

  • I was actually just about to make a note of the existing proposal to allow extra close votes in /review, once I can find that, and note further that the scaling here could add e.g. +2 review-only votes for every 10 reviews added to the cap. – Nathan Tuggy Oct 26 '15 at 19:26
  • 1
    Just for bullet 2: douze points – rene Oct 26 '15 at 19:26
  • 2
    For your "... the close votes need to scale with the increased review limit too. One idea could be to split the on-site close votes from the review votes.", consider Give extra close votes, only accessible via /review – user213963 Oct 26 '15 at 19:30
  • 1
    Good one. Upvoted! @MichaelT – Patrick Hofman Oct 26 '15 at 19:31
  • 1
    @MichaelT: Yes, that's the one I was thinking of. – Nathan Tuggy Oct 26 '15 at 19:37
0

Reminds me of the AIMD algorithm. In TCP/IP you increase the rate of packets incrementally (additive increase), but when one is dropped you divide the rate (multiplicative decrease) so you are essentially hunting around for the best transmission rate without loss of efficiency caused by packet drops.

This algorithm won't work directly applied here (At minimum it needs to be reversed, and it has problems at extremes), though perhaps a variant will, however the point I want to make is that we probably shouldn't keep "fixing" this problem over and over and over again. We should consider algorithms that will take into account the amount of users actively participating, the average queue sizes, and the average amount of help each user is putting in.

We don't want to go to an extreme where a large queue gives one person a huge amount of votes, but we're already suffering under the current "fixed" quotas which aren't really helping on the sites they're meant to help.

At any rate, I don't think the minor change suggested will prove a successful long term solution that won't require further thought and tweaking later.

  • 1
    Can you explain better what problems you see in the suggestion that are likely to require fixing later? Any idea might later turn out to need refinement, in principle, so declining to make any changes on the basis of unspecified or unknown possible future problems does not seem particularly workable. – Nathan Tuggy Oct 26 '15 at 20:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .