Can we raise the bar for reputation for late answers to bypass the review queue?

Question

I would like to raise the bar for late answers to enter the Late Answers Review Queue to 50 rep, which is the threshold after which users gain the ability to comment. Here are some answers that were posted by users with between 10 to 50 rep (yes, there's a little selection bias to be sure. I suspect most/all of these will be 10k links only by the time you're reading this. Note, all of these are from the last 5 hours.

These are all from Stack Overflow, where I have the most experience, but since Late Answers have a much higher probability to be bad across all stacks, I think this applies generally. Also, I have bolded all the users references to the inability to comment, that were not in their original answers:

• https://stackoverflow.com/a/32702471/1768232 (23 rep)

  Solution provide by @brandonyang works for me too! (I can't add comments in a response 'cause my low reputation, sorry).

• https://stackoverflow.com/a/32703040/1768232 (47 rep)

  I tried your answer but it doesn't work in office 365 outlook online. Do you have any suggestions?

• https://stackoverflow.com/a/32701373/1768232 (26 rep)

  My question is somehow related to the same issue. I am fairly new to C# and have been doing JAVA for quite a long time. When I am coding in C# with .Net framework 4.5.1 using visual studio 2013, I get errors for using certain libraries but I donot know which dll should I reference in my project How do C# people figure that out? For example I wanted to code to draw an ellipse and needed system.Windows.Media and System.Windows.Controls both of which were giving me errors, after googling for quite sometime I got to know that we need to reference PresentationFramework.dll, is this the only way to figure it out?

  PS >> I am unable to post this as a comment hence putting it as answer.

  Thanks.

• https://stackoverflow.com/a/32699722/1768232 (19 rep)

  i like this video. It explains the differences between int and Integer and cautions of using Integer

• https://stackoverflow.com/a/32698255/1768232 (21 rep)

  I have a question, i am actually trying to copy the database from Prod server to test server ( SQL server 2008R2). I am wondering would it copy the users logins, roles and permission as well ?

---

I know I suggested 50 rep above, but while I was compiling that, I found one from a 61 rep user. Perhaps it should be 100 rep, not 50. I am less confident about this, but I thought I'd add it as an addendum for discussion.

The answer posted here is the simplest solution that you are looking for to solve your problem, so take a look ;)

Answer 1

TL;DR: As of today (September 29, 2015) the maximum reputation for having an author's late answer enqueued has been raised from 10 to 50.

As for further increase to 100:

Queue growth

Review queues lose effectiveness if posts aren't regularly (and accurately) cleared. A queue such as close votes on Stack Overflow that never seems to get to 0 fails to provide time-sensitive feedback that aids learning. So if we raise the bar for late answers which increases the number of tasks going into the queue, we risk making the queue less useful for people answering questions late.

As I write, the Late Answer queue on Stack Overflow is empty. In the last 24 hours the number (non-audit) review tasks of each type added to the queues:

Review Tasks Type           
------------ ----------------- 
2467         Triage            
2428         First Post        
2333         Close Votes       
1846         Suggested Edit    
 883         Low Quality Posts 
 619         Late Answer       
 594         Helper            
 141         Reopen Vote    

These two factors suggest that there's room to add more tasks to the Late Answer queue. Assuming we raised the reputation limit to 100, we'd add about 300 tasks:

Rep Bin Tasks 
------- ----- 
 10       417   
 20       160   
 30        85    
 40        76    
 50        44    
 60        41    
 70        37    
 80        21    
 90        20    
100        20   

You might expect the number of tasks associated with the 1 to 10 reputation bin would equal the number of late answer tasks in the last day. The discrepancy stems from reputation earned since answers were added to the queue. But the takeaway is that raising the reputation limit to 100 would add half again as much work to the queue. That doesn't seem particularly onerous.

Not surprisingly, Stack Overflow would see the most added Late Answer reviews by far. Other large sites, such as Ubuntu, Server Fault, Super User and Mathematics, would get 10 or so extra reviews a day.

Justifying 100 rather than 50 reputation

Discovering answers as comments certainly would suggest 50 as a good limit for this queue. But this queue is as much about granting exposure to potentially upvote-worthy answers as it is about finding problems. Especially on large sites, it's pretty easy for a good answer to a very old question to be lost in the noise. As a strategy for new users answering old questions that already have good, but not excellent, answers seems productive if not that few people will notice those answers. At 10 reputation, a knowledgeable new user has one shot at earning upvotes from excellent answers to aged questions.

A potential complication

The one concern I have is that answers are dequeued once the author has reached 100 reputation. If we raise the maximum reputation for going into the queue, we should also raise the minimum for coming out of it. Unfortunately, I don't have an easy way to gather numbers to set an appropriate level. One of the reasons I'm only looking back a day in the numbers I showed above is that it minimizes the effect of gained reputation. It's possible, but messy to estimate reputation at the time a user answered a question. So while my instinct would be to raise the deque minimum to 200, but I can't really predict if that's appropriate or not.

Resolution

Raising the bar to 100 would require changing two variables at once. Since that would make it harder to evaluate each individual change, I raised the reputation for getting a Late Answer enqueued to 50. I will reconsider upping it to 100 at some point in the future. (Hopefully we can do it in a less jarring manner, however.)

Answer 2

I think this is a good idea. I spend a lot of time on the New Answers to Questions More Than 30 Days Old list on Stack Overflow (10k link), and I would agree that many of the questions I flag as not an answer are from new users who want to comment, find they don't have the privilege, and leave an answer instead.

The main concern I would have about the proposal would be whether this would overload the Late Answers Queue. To analyze that for Stack Overflow, I've looked through the 200 most recent late answers on the site. Here is the breakdown by reputation of the poster:

• 45 (23%) were from users with reputation 1-10
• 46 (23%) were from users with reputation 11-50
• 10 (5%) were from users with reputation 51-100
• 99 (50%) were from users with reputation 101+

From this limited dataset, it seems like we might see roughly a doubling of the number of answers added to the Late Answers Queue by increasing the reputation limit to 50. Increasing to 100 would not add too many more late answers to the queue. Given that the Late Answers Queue is often empty on Stack Overflow, I don't think this would be too problematic.

To get a sense of the users who are creating problematic late answers, I reviewed my most recent 200 helpful flags on late answers on Stack Overflow. Here is the breakdown by reputation:

• 132 (66%) were from users with reputation 1-10
• 39 (20%) were from users with reputation 11-50
• 10 (5%) were from users with reputation 51-100
• 19 (10%) were from users with reputation 101+

This suggests that users with reputation 1-10 are generating much more than their fair share of problematic late answers, users with reputation 101+ are generating much less than their fair share, and users with reputation 11-100 are generating about their fair share. I think this could mean that reviewing late answers for users with up to 100 reputation could make sense, but there would be diminishing returns raising the bar above 100 reputation.

It would be helpful to see this analysis repeated on other sites to see if the results are qualitatively similar (it sounds like the relatively small size of the Late Answers Queue may hold across other sites).

Answer 3

Perhaps letting people comment sooner would also be a way to solve this problem. Note that many of the cases cited here are people w/o the rep to comment. Creating a situation where they are thus "forced" to post an answer that isn't an answer is actually not a good thing. It violates the "truth in packaging" idea; makes the user (presumably usually new) feel bad about the whole site, and invites others to criticize their posting an answer that is not an answer, thus cycling the whole thing downwards... being a new user on SO is not easy (and I happen to be a PhD who had about 15 years of industry experience when I started using!)