I've been putting this off for a long time...
Close vote aging - the deactivation of votes that haven't resulted in a question being closed - is a critical part of the vote-to-close system, but has something of a troubled history: all too often, it has been more annoying than useful, capriciously blocking the closure of awful-but-obscure questions. We've spent the past couple of years experimenting with changes to make the rest of the system operate more effectively, but now it's time to bite the bullet and hammer this last piece into place.
My proposed change is simple - skip to the last section here if you wish; I know you're all tired of reading my overly-long essays about obscure parts of Stack Exchange, but whether you realize it or not this poorly-understood system has a profound effect on how these sites operate. Therefore, I humbly ask that you bear with me as I try to explain the rationale behind this proposal.
The importance of vote aging
Closing can be both fair and effective when a question is quickly brought to the attention of multiple trusted users, who may then opt to vote or to refrain. Problematic questions can be shut down quickly or fixed if possible, and discussion concerning problems with the question can be conducted.
But when this doesn't happen, things get dicey: problems may go unaddressed and the direction of moderation may be unfairly influenced by a subset of privileged users. One can easily imagine situations where, given sufficient time, any question might be closed due to the gradual accumulation of votes from a tiny minority of voters, and indeed this does happen.
Vote aging is designed to minimize this, by creating a ticking clock that demands a decision be made promptly and otherwise considers the matter resolved. But after a couple of years, a problem with this system became apparent...
The rationale behind the current rules for aging
These are the current rules for vote aging:
when the aging task fires it just looks at every post that has at least 100 views where all outstanding close votes are older than 4 days.
Then it deletes a close vote. A day later, it runs again and does the same thing, until either another close vote shows up, or all the close votes are deleted.
Simple enough, right? Except for that "100 views" thing. There is a long tail of questions on these sites that take months or years to garner that many views; sure, some of them are awful, but there's also a treasure-trove of hard-to-find information among them - surely lack of popularity isn't a good reason to damn a question?
This requirement exists because back in 2011 it was getting really hard to get enough people looking at lousy questions to close them. Review didn't exist yet; there were 10K tools to aid in reviewing close votes, but not enough people using them to really matter. And folks were getting frustrated; in many cases, it was more effective to flag and ask a moderator to close than it was to vote! The community of trusted close-privileged users was becoming ineffective; the bulk of closing was being done by moderators, acting at the behest of anyone who could flag.
The change in how aging works worked - at least, it kept votes around long enough for someone to act on them. But it didn't really address the underlying problem, and it made vote-aging considerably less effective.
Figure A: Close vote results over time
Here's a chart of close votes by month, along with some annotations for various events. Unfortunately, we hard-deleted close votes before the summer of 2011, so I don't have accurate data with which to illustrate the problem that this 100-view requirement addressed - you'll have to just trust me that it was a real thing, and that this did help. I can demonstrate how it failed to address the underlying problem though...
Figure B: Close vote effectiveness by question view-count, summer of 2011
That chart needs some explanation. The X axis is views, the Y axis is the percent of close votes that resulted in closing (5th close votes and moderator closes are excluded here, as they're instantly effective 100% of the time). Each line represents a timeframe: 4 days, 16 days, 30 days, 60 days and all time. The data reflects the four months following the addition of the 100-view requirement for vote aging, but the actual view counts are current.
The key observation here is that the 100-view aging threshold made time irrelevant for the questions that fell below it: there's very little difference in the effectiveness of close votes after 4 days and 60 days, and not much more for all time. But once that 100-view threshold is past, effectiveness drops like a rock...
An obvious question here is why the all-time effectiveness for even questions over 100 views is pretty high: well, remember, I'm looking at the view count now - a big chunk of the questions closed at over 100 views started out with less, and didn't garner more until after they'd been closed.
And that's where Review comes into play...
How Review affects this
Refer back to Figure A in the previous section. See where the new review system was introduced? Suddenly, we didn't have to rely on chance or even personal taste to give votes nominated for closure a look from privileged users. Close votes - and actioned close votes - exploded! Now compare Figure B to...
Figure C: Close vote effectiveness by question view-count, summer of 2014
It took us a couple of years to work out all the kinks; scaling up something like this presents some rather unique challenges... Didn't help that we changed all the close reasons half-way through: we suddenly had a LOT more people reviewing a LOT more questions with unfamiliar and occasionally confusing options. And the old aging rules still in effect... The result was chaos: instead of benefiting more obscure tags/topics, the views threshold ensured that the smallest problems became the biggest clogs in the pipeline, throwing them up for review again and again and again until they would finally get closed - meanwhile, questions that garnered lots of organic views dropped out fast no matter how problematic.
We had built and refined a vast system for democratizing justice... and then put it to work enforcing parking tickets.
Isn't this just a Stack Overflow problem?
No. Actually, right now this is less of a problem on Stack Overflow - a few months back we turned on a system for prioritizing reviews there, something of a twist on the old aging scheme: just kicking stuff out of the queue when it clearly wasn't making any progress, focusing the attention of reviewers on areas where they stood to do the most good and least harm. That... Actually worked pretty well:
Figure D: percentage of initial close votes actioned in Y days, by month
The summary of this chart is: 90% of questions that get closed on Stack Overflow now get closed in under 2 days; 50% get closed in less than an hour!
This isn't true for other sites though. Here's Figure D for Ask Ubuntu:
Figure D.AU: percentage of initial close votes actioned in Y days, by month
And remember Figure A? Here's figure A on Ask Ubuntu:
Figure A.AU: Close vote results over time
You see the same problem developing there as on Stack Overflow: a growing number of unactioned close votes that aren't aging. We could try to follow the same path there as on Stack Overflow, prioritizing the close queue - but that's a bit of a band-aid (it still doesn't avoid votes trickling in organically over extremely long periods of time), and has to be heavily tailored to the site. Also, there's one more problem with aging...
You got ONE SHOT - better make it count...
Remember the original problem that the 100-view threshold was intended to address? Well, what happens today if you find a seriously problematic question in the obscure tag that you follow, you vote to close it, and... Nothing happens. Maybe your vote hangs around forever, or maybe enough random viewers trickle in from Google to hit the view threshold and your vote ages away.
What can you do? Not much. You can't re-cast a pending vote, and you can't re-cast an aged vote. Your proposed closure was put before the committee and... ignored. Maybe there were a lot of parking tickets that day.
Some folks raise a flag when this happens, but moderators are often reluctant to intervene on topics they aren't personally familiar with unless the problem is truly egregious - they signed up to be exception-handlers after all, and a question no one cares about isn't all that exceptional. Others lean on the Very Low Quality flag to give questions a second shot at review - but strictly-speaking, this is an abuse of VLQ and will probably get harder as we continue to develop review.
What you'd like to be able to do, upon coming across a problematic question you've previously voted to close, is just bump it back into the close queue. Once upon a time, this was actually the recommended way to handle vote aging on questions if you kept tripping over them - but at some point, it was disabled, probably because folks abused it to keep harping on some personal annoyance.
Proposed changes
Finally, we're here. That bit above was hard; this is simple:
Start aging votes after 14 days, even if the view count is too low. That is, if the newest vote on the question is 14 days old, aging will begin even if the view threshold has not been exceeded. If the view threshold is exceeded sooner, then the existing time threshold (4 days since newest vote) will be used instead.
This should be site-configurable on the off-chance that we need it to be greater somewhere (or decide that even this is too long to wait), but realistically this should be sufficient - the vast, vast majority of questions that do get closed are closed in under half this time. We should keep the view threshold in place for now just in case I missed something crucial here, but should plan to reduce it to some token number in the near future - just high enough to protect against insane edge-cases that get missed entirely, but not so high that it's in effect on a significant number of questions.
Allow re-casting votes that've aged away after 14 days. That is, 14 days (use the same site-configurable value used in #1 here) after your vote has aged away, you're free to cast the same vote again.
If your first try doesn't get any attention, and you keep tripping over the same problematic question... You should get to have another go at getting it reviewed. A 14 day waiting period means you don't get to spam reviewers with every question you personally don't like, but does give you that opportunity to correct oversights without abusing flags for the purpose.
For consistency, both of these changes should be applied to Close Votes, Close Flags, and Reopen Votes.
That's... It. We discussed lots of crazier, more complicated ideas over the past couple of years, but kept coming back to the idea that we were fixing the wrong problem - and indeed, we were. When Review works, vote aging can be pretty simple; when it doesn't, there's a deeper problem that needs to be fixed. At last, it's time to rip off the adhesive bandage and see the moisting flesh underneath...
