20

Suggest to setup a dedicated queue for moderators to review questions that hang in Stack Overflow CV queue for more than specified amount of time - say for 5, or 10, or 20 days, or maybe for a month.

  • For this queue, suggest to have no review limits set (as moderators are expected to be trusted), allowing them to pick the comfortable rate at their discretion. Also I am not sure if audits or badges make sense there.

Taking into account absence of review limits, along with the fact that per my observations, moderator vote is binding in the sense that both Close and Leave Open drop the questions off the "main" CV queue, I would expect this to help in handling "long tail" of the CV queue.


Above assumes that moderators are interested in helping community to handle CV queue. This assumption is in turn based on my observations, for example of current top reviewer being a moderator with fairly respectable 12K+ reviews. Also, I noticed that several dozens of my flags bringing mod attention to particularly tricky cases in CV queue at Programmers were actioned with an impressive consistence (in comparison, other kinds of my flags result in no action from time to time).

For a somewhat similar suggestion, take a look at Let moderators navigate the review queues at will - mine is different in that suggested queue is designed to pick and serially present particular kind of troublesome items for mod attention.

14
  • 1
    hmm given that question currently has 5 tags, wonder which one will be thrown out if it additionally gets one of the status-* flags?
    – gnat
    Sep 27, 2013 at 7:56
  • 3
    Reading your posts takes too much time as it contains too many (hidden) URLs.
    – Himanshu
    Sep 27, 2013 at 10:56
  • An interesting question would be how many questions this would be; I fear it might be a lot! Sep 27, 2013 at 10:58
  • @RichardTingle I would rather spell it as how to tune this queue, meaning how to pick a "time cut" that would show a sensible amount of questions - along with how to define what amount is sensible
    – gnat
    Sep 27, 2013 at 11:01
  • 1
    This could certainly help with the lost flag issue that arose form the change that sends flag-to-close into the CV queue instead of the Mod/10K queue. That is the issue of flags getting lost in the queue and since they never expire, they could remain active forever. Sep 27, 2013 at 11:26
  • 2
    @hims056 did you notice that links have tooltips?
    – gnat
    Sep 27, 2013 at 11:28
  • 2
    @gnat - I am used to read tooltips of your links. You may not believe, but actually I learned it from you. :)
    – Himanshu
    Sep 27, 2013 at 11:30
  • 2
    Hmmm... interesting idea. It could sort of be seen as just a different way to sort the existing queue, maybe? And perhaps available not just to diamonds? Sep 27, 2013 at 16:07
  • 1
    Given that there are 80k+ questions in the CV queue, presumably most of those will be older than a month (or however long is decided upon), so we'd just be rubbing off questions on the very few mods, which doesn't seem like a good idea. Though a separate queue might not be a particularly bad idea because, since "moderator vote is binding" they can be given mainly the 1-vote questions which they can instantly close, rather than being given 4-vote questions, which would have their vote be somewhat wasted. But leaving open a 4-vote question would presumably be similar, so I don't really know. Sep 27, 2013 at 20:40
  • @AndrewBarber yeah when preparing this request I was thinking of this "variation", too. Trusted reviewers and such, sort of "sheriffs". Left it out because it looked more complicated. For diamonds, all the stuff looks just... natural. Yeah, they are trusted for unlimited reviews. Yeah, their vote drops items out of the queue. Yeah, it doesn't feel weird for them to have no audits, fine. How could regular users be promoted that way? could they? it's somewhat complicated; I'd rather first give it a try with diamonds, and only if this turns insufficient, consider widening an access to the queue
    – gnat
    Sep 27, 2013 at 21:22
  • 1
    Good point... I wasn't thinking of those extra points about it. Sep 27, 2013 at 21:24
  • @Dukeling I was estimating their capabilities based on my own experience. Per my recollection, I typically do 40 CV reviews in about 10-15 minutes. I think I could do about 200 (maybe 400) reviews a day with ease, now let's calculate. 200 reviews, 5 days a week, each one drops the item off the queue. 1 week, 1 user => 1000 items off the queue. 5 users, 4 weeks => 20K items off it. 4 months to drain current 80K, simple math.
    – gnat
    Sep 27, 2013 at 21:27
  • 1
    @gnat You're not taking into account inflow (unless you assume this is balanced out by current non-mod reviewers) and assuming 5 mods will spend like an hour a day, every day, reviewing, but it really doesn't look like they do (I know I wouldn't want to). Oh, and can mods review 400 questions? Sep 27, 2013 at 21:38
  • @Dukeling right, I am intentionally not taking inflow into account... and neither taking into account community reviews, did you noticed? You are right that mods (except for Bill) don't focus much on CV reviews, but this is because as of now, it is considered mostly community job. As for 400/day, I wouldn't aim at it, at least not in theory. 200 feels sane... and even this estimate would better be tested to make sure
    – gnat
    Sep 27, 2013 at 21:42

1 Answer 1

7
+100

This isn't a bad idea...

However, one of the big reasons for taking close flags out of the moderator queue was to get moderators out of the business of mechanically closing hundreds of questions each day. The whole point of voting to close was to attempt to establish some consensus as to which questions are appropriate; if a sizable number of questions are being closed with a single vote anyway then we might as well abandon that whole mess.

So I wouldn't define "stuck" as simply "has been in the queue for a while". Sure, maybe that means a really terrible question has been overlooked... Or maybe it just means the community doesn't really consider it much of a problem at all. Tying up the moderators with the latter isn't doing anyone any favors.

We ended up doing something a little bit different for the Low Quality queue: if a post goes through review with no action but later gets another flag, it ends up in the mod queue only, effectively escalating the flag. Something similar might work well for close reviews. A few ideas:

  • Promote questions that get 5 votes with 4 aged away to the moderator queue, then crank up the rate at which votes get aged (drop views requirement, reduce # of "do not close" responses needed, etc.)

  • Dequeue tasks that've been in the queue for more than a few days with no reviews or votes, but don't discard the reviews or votes. Reactivate them if another vote or flag is raised.

  • Require a minimum number of votes or flags for posts to even enter the queue in the first place (right now, that number is 1 - it could be 2 without seriously disrupting most closures, and even higher if paired with some other trigger).

  • Fix the moderator flag queue such that moderators have more time to participate in the normal /review queues.

Thoughts?

9
  • 1
    What's happened to the "How can we improve the mod tools" idea? That question was asked in July by Laura, and I can't think of any ideas that's been implemented yet?
    – Matt
    Dec 2, 2013 at 17:53
  • 2
    At least one of them (q-ban status) has been implemented, @Matt - there were more scheduled (comment locks being a big one) but we've been extremely resource constrained for pretty much the entire summer/fall and just haven't been able to get to any of them.
    – Shog9
    Dec 2, 2013 at 17:56
  • 1
    4 votes aged away -- this way to define "specified amount of time" suits me well. Also, narrowing it with additional requirement to have 5th vote makes perfect sense (it feels fair that 4 lone votes plainly aged away don't warrant mod intervention). Other points also look good, except for one about dequeuing - this one may also be good, I only need a bit more time to chew it
    – gnat
    Dec 2, 2013 at 18:02
  • 1
    Yeah, so do I, @gnat. It's easily the most complicated of the four, and frankly I'm having a bit of trouble simulating the effects - I think it's something we'll need eventually, but probably worth trying other things first. Got another post that elaborates on that a bit more in the works.
    – Shog9
    Dec 2, 2013 at 18:04
  • 2
    queue UI sketch: i.stack.imgur.com/4mHqM.jpg (courtesy of Pëkka)
    – gnat
    Dec 13, 2013 at 18:33
  • @Shog9, please can you point to your other post. Feb 17, 2014 at 16:37
  • Which post, @Ian?
    – Shog9
    Feb 17, 2014 at 18:30
  • @Shog9, the one you siad you were about to write in your comment of 18:04 on 2nd Dec 13. Feb 17, 2014 at 19:06
  • Yeah, that's currently held up by one of the other items in the list which has been in dev limbo for 2+ months. Not much point in adding to the noise until we have some of the low-hanging fruit picked. @Ian
    – Shog9
    Feb 17, 2014 at 20:23

Not the answer you're looking for? Browse other questions tagged .