25

Was there a recent change in the way the first posts review queue works? I ask this because I am used to getting the reviews from that queue within the first five minutes after the question (or the answer) having been posted. But, since yesterday, it seems that a post appears in the queue only about an hour after having posted. I noticed this at Mathematics Stack Exchange, and also (but I am not so sure about that) at TeX Stack Exchange.

Example with 37 minutes difference:

enter image description here

11
  • 4
    I've noticed the same thing on English Language & Usage, in some cases the reviews only appear after a couple of hours and they often arrive in batches. I suspect that the process that detects first posts is now running less frequently. – KillingTime Apr 15 at 10:20
  • 1
    @KillingTime Yes, they do arrive in batches. I noticed the same thing too. – José Carlos Santos Apr 15 at 10:21
  • 5
    It's not just First Posts; Close Votes are affected too. This question should've had a review item created 15-20 minutes after Spevacus' comment, but it didn't. – Glorfindel Apr 15 at 13:54
  • 4
    This appears to be getting worse. This example from ELU has a question added to the 'first post' and 'close' review queues almost 6 hours after it was originally posted. If the aim of the first post review queue is to get a quick response to users' first posts then the system would appear to be failing in that aim. – KillingTime Apr 20 at 17:23
  • 3
    The same thing happens to Academia.SE. Very likely it's SE wide change. The change actually defeats the purpose of Review. We want to review posts as soon as possible so bad posts would be found in time and then report to the community/mods (by flagging or other means). Now, this change delays the bad post reporting. Does whoever made the change understand this? – scaaahu Apr 21 at 2:41
  • 2
    Something has changed. I just got a question on the first posts review queue two minutes after having been posted. – José Carlos Santos Apr 21 at 6:17
  • 2
    I can confirm that we are indeed seeing a regression in the time it takes for posts to enter review queues. Review tasks are synced (created/cleared) in batches every 5 minutes but batches can be skipped for performance reasons. On April 13 we started started seeing a significant number of skips and we're investigating the cause. – Brian Nickel Apr 24 at 0:43
  • @BrianNickel Thanks for the confirmation. The issue has been unstable in the past couple of days or so. For a while, I thought it's been fixed. But, then it seems that the issue comes back again. I have reviewed more than 1k Late Answers on Academia SE, my experience was that it took 15 minutes to enter the queue if the answer is okay, 18 minutes to enter the queue if someone raised LQP/spam flag flag. Last time, I believe it took more than 20 minutes before I had chance to see it.. – scaaahu Apr 24 at 6:39
  • 1
    @BrianNickel The OP of this question has 137 Steward badges on Math SE, which means they reviewed more than 137,000 posts on Math SE. I have 20 Steward badges on Academia SE, the most number of Steward badges on that site. I cannot speak for others, let me speak for myself. This change weakened my morale on SE. Please don't let us down. Thank you. – scaaahu Apr 24 at 6:43
  • 1
    Just a mid-sprint update, but it does look like the issue went away on its own like @JoséCarlosSantos reported on April 21, with no clear cause. My suspicion is that we had more work to get done in 5 minutes than could get done and our queuing system turned into a free-for-all between new and old jobs. I'm looking at getting monitoring around this and a few simple mitigations. – Brian Nickel Apr 28 at 21:51
  • 1
    @BrianNickel For giant size SE site like Math SE, your explanation makes sense to me. For mid-size site like Academia SE, it doesn't make too much sense. At low activity times, we (Academia SE) gets only a few posts in several hours. So, I am not convinced by your explanation. – scaaahu Apr 29 at 14:39
6

This problem fixed itself for now. We're not really sure what the precipitating factor was but I have my suspicions around what happened and plans to address it.

Some background

With the exception of suggested edits and the triage queue, review tasks don't instantly populate the queue when their criteria are met. This is a design as old as review queues and is probably driven by the fact that most tasks don't land in the queues until 15 minutes after they otherwise qualify.

Every 5 minutes, every site runs a job that:

  1. Finds every post that should be in a review queue but isn't.
  2. Finds every post that is in a review queue but should no longer be.

It then creates tasks for every post in 1 and deletes/invalidates every task from 2. These queries have bounds and performance tweaks but can still be very hard on SQL Server.

At the same time, we have a couple other jobs that can be hard on SQL Server, the awarding of badges and an hourly syncing job. At the top of the hour, we could have 1000 of these jobs all wanting to run at once.

Since we have all our sites run on shared infrastructure, we can't have 1,000 jobs all simultaneously trying to get to run their expensive queries, so they get queued up to only run 8 at a time. This keeps the network from falling over.

So what probably went wrong?

My suspicion from looking a differences in per-site behavior, is that we simply had more work to do in 5 minutes than could get done and scheduler wasn't designed for that. When each job needs to run, it attempts to grab at a semaphore lock to get permission. By its nature, the semaphore isn't FIFO but closer to random chance as to who runs when. That is fine under normal conditions because there's not a huge difference in quality of service between being the first and last job to run.

Under our supposed conditions, where maybe 50-100 or more jobs are waiting to run after 5 minutes, we run into a big problem. The jobs that just got to run are now queueing up again and not waiting patiently behind the tasks that got there first. Random chance could keep a job waiting for minutes or hours.

Anyway, that's my theory.

What are we going to do about it?

I had a lot of proposals that got reviewed, rejected, and tweaked that I'm not keen on relitigating, but I can tell you what our short-term and longer-term plans are here.

In the short term, I am setting up monitoring so we can see how frequently these jobs are running or failing to run and alert if there's a major disruption. Hopefully prior to you observing it. If we do observe it, we can temporarily decrease the frequency of jobs to give them the breathing room they need to complete.

In the longer term, our plan is to make infrastructure improvements that move from our more expensive syncing queries to event based task creation and invalidation. That will eliminate the sync query overhead and risk and simplify workflow changes we already had planned. I'm already working on a POC for First Posts since it's the least complicated of the bunch.

Incidentally, this was already on our radar for the project but this incident and meta report has bumped up the priority.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .