Say a reward was offered for a machine learning system that could predict the outcome of part completed reviews, and a nice test dataset was provided, would it make a interesting project for 3rd year (or master) Computer Science students?
Then if a system was found that would predict the outcome of some reviews 99% of the time, it may offer a way to allow these reviews to be auto completed.
For example, it may be the case that a review that is skipped by a lot of users, and “leave open” by 2 users, without a close vote cast, can just be auto completed on the bases that it will hardly ever result in the question being closed.
Some more thoughts on this.
I think we have 3 “sets” of questions reviews.
- Bad: Questions that clearly should be closed.
- Good: Questions that should be clearly left open.
- SoWhat: Questions that are of little value, but not so bad as to get into the “Bad” set. Do we care what the outcome is on these?
I expect that most review tasks are in the “SoWhat” set, and that most people will agree that review tasks in the “Bad” set are more important.
So if there was some way to predict if a review task was likely to be in the “bad” set, it could be moved up the review queue so it got completed quicker.
If it could be predicted that it was very unlikely that a task was in the “bad” set, then it could just be removed from the review queue. (It will go back in if it gets anther close votes.)
I expect that by looking at how long reviews take, how many skips, the past history of the reviewer etc, each review would be found to provide more information on the outcome, then is just picked up by looking at the number of “close” and “leave open” votes.