Late late answer?

Question

So, I just came across a review in the Late Answers review queue.

The link to the review itself: https://softwareengineering.stackexchange.com/review/late-answers/47599

This wasn't an audit (the dates on it struck me - I went out to double check that the vote count matched). Its a late answer from two years ago (on a three year old question) showing up in the late answer queue.

Is there something amiss from scripts? or is this intended functionality that old, unreviewed, first posts occasionally pop up in the queues?

I blame caching.

Answer 1

This is, while unusual, an expected result of the system.

The way the late answers queue works is by checking two things.

• Was the answer posted at least 30 days after the question was asked?
• Was the answer posted by a user who has 10 50 or less reputation (disregarding suspension)

If both of these are true, it is entered into the review queue until it eventually gets reviewed. In most cases, this will catch things as they come in and so you will only see new posts, so we don't spend any resources checking time.

However, on rare occasion, someone will actually drop below 10 reputation, while having some past answer that does qualify. That's what happened in your example - the user received a dunk in reputation from 15 down to 5 on December 21st, 2013, and so this ancient post of theirs that has not previously been reviewed, is now put into review that same day.

Answer 2

This has been a regular source of confusion (and occasionally, excessive amounts of work) for years. After a thorough analysis, I'm unable to find a compelling justification for this behavior; "late" late answers are infrequently reviewed, and even more rarely actioned; the exceptions are primarily on sites where the queue piles up for weeks and then gets cleared out, mostly negating the value of having a queue in the first place: giving feedback to a new user weeks after they've posted has little utility, while tools like the (10K-only) "New Answers to Questions More Than 30 Days Old" are much faster to scan and more thorough in their criteria for inclusion if you're just looking for non-answers.

So with that in mind, today we're gonna start excluding these answers from the queue once they're more than 7 days old. This is how it has worked on Stack Overflow for the past 6 years already, so there's a side bonus of consistency here... However, this threshold can be adjusted on a per-site basis if necessary.

While I was reviewing the data for this, I also took a look at the First Posts queue: posts here were dropped after they were 30 days old, but most actioned reviews happened much more quickly; here the exceptions were limited to sites where the majority of review tasks were invalidated, either because the post was closed or deleted from outside the queue, or because it simply got too old. Again, not much value in letting these pile up: it becomes discouraging for reviewers and provides progressively less value to either the author or the site. So I'm reducing the threshold here to 14 days, which covers the vast majority of actionable reviews.

...I also managed to break Late Answer syncing for a solid hour, so... Sorry about that.