Scale daily review caps more effectively to bring even busy queues to 0 instead of 1000

Question

On a number of sites*, there's enough review flow to reliably keep certain queue backlogs in the dozens or even hundreds. This is discouraging to regular reviewers; some sites struggle recruiting more than a few reviewers, and a number of comments have cited large queue lengths as the demotivating factor limiting their own involvement. It also increases time to handle new items, making community moderation less effective and causing more work to clean up e.g. answers to questions that are not closed promptly.

The only feature that currently acts to actively restore the balance in most such cases is the raising of daily caps to 40 instead of 20. But because this takes effect only when the queue has a backlog of 1000 or more, and cuts out as soon the backlog drops below that (without waiting for any user to finish with their expanded allowance), it does nothing to actually bring the queue to 0. Clearly, though, the general approach is considered philosophically acceptable: if there's just too much in the queue, give each willing reviewer somewhat more opportunity to try to take care of it all, even at the expense of slightly reduced accuracy or higher per-reviewer load.

What should be changed, therefore, is the trigger. How about a graduated system? For any week** in which a given queue never reaches 0, the daily cap for all users for the next week** is raised by 10, up to a maximum of 50. Conversely, for any week** in which a queue never reaches 20 items backlogged, the cap is lowered by 10, to a minimum of 10. If neither condition is met, the cap remains the same as the previous week**, and caps are initialized to the result of the current calculations. Transitions can be very efficiently tracked by checking whether a given queue is at 0 or 20+ whenever the 10k-tools review count is updated, and setting a bit field if so.

Some queues are limited by votes, not just the review cap: Close, Reopen, and certain functions of Low-Quality. These should ideally allow additional queue-only votes; specifically, allow 5 review-queue-only close votes for every 10 reviews in the cap (including the baseline). So a queue operating at 50/day/user would allow the first 25 close votes to be cast for "free". (These would probably be shared between all eligible queues, including Triage on SO, since the alternative requires a much more complicated UI.) Delete votes for 20kers in LQP should be handled similarly, but the scaling would need to be rethought; that deserves a separate proposal.

The primary benefit of this proposal is twofold: queues that are not absolutely incorrigible could be brought reliably to heel by the community, removing the futile perception of fighting perpetually multi-hundred-item queues, and newly busy queues would be adjusted for much faster, reducing system lag in most cases of increased load. (Not counting events like the recent Late Answers surge, of course.) Fundamentally, this addresses the key problem of review queues, which is not current backlog, but capacity as a rate at the margin.

The general principle behind this idea has been recently tested at SO, and the results were officially promising enough to warrant some sort of scaling increase. I think this algorithm is one of the best choices for this: it's relatively simple but quite flexible anyway.

_{*SO's CV queue is of course legendary, and would not be materially affected by this, but the recent growth of LQP there seems to have no real end in sight (900 at time of writing, and still climbing fast). Other sites affected include SF and SU, ELU, Android, DIY, AU, and probably dozens of others I'm less aware of.
**Or day. I'm not sure which granularity is likely to be better.}

Answer 1

This is an excellent idea, but it has some concerns too that need to be addressed before this can be effective.

1. Scale close votes, etc. too. In order to be capable of doing anything in the close or reopen queue (and others, just as an example), the close votes need to scale with the increased review limit too. One idea could be to split the on-site close votes from the review votes. This means we don't need to give users 100 close votes, but just their regular amount plus 100 review close votes, the exact amount of the rate limit of reviews on that day. Effectively this means that close votes from the review queue won't count to your day limit any more.

2. By increasing the number of votes, the effect can be that even more users skip reviewing since others do it already. I like the part of the community that does their review job every single day, but this could put extra load on them instead of others. The question is: how to keep motivating other users to do their review jobs?

Answer 2

Reminds me of the AIMD algorithm. In TCP/IP you increase the rate of packets incrementally (additive increase), but when one is dropped you divide the rate (multiplicative decrease) so you are essentially hunting around for the best transmission rate without loss of efficiency caused by packet drops.

This algorithm won't work directly applied here (At minimum it needs to be reversed, and it has problems at extremes), though perhaps a variant will, however the point I want to make is that we probably shouldn't keep "fixing" this problem over and over and over again. We should consider algorithms that will take into account the amount of users actively participating, the average queue sizes, and the average amount of help each user is putting in.

We don't want to go to an extreme where a large queue gives one person a huge amount of votes, but we're already suffering under the current "fixed" quotas which aren't really helping on the sites they're meant to help.

At any rate, I don't think the minor change suggested will prove a successful long term solution that won't require further thought and tweaking later.