25

Say a reward was offered for a machine learning system that could predict the outcome of part completed reviews, and a nice test dataset was provided, would it make a interesting project for 3rd year (or master) Computer Science students?

Then if a system was found that would predict the outcome of some reviews 99% of the time, it may offer a way to allow these reviews to be auto completed.

For example, it may be the case that a review that is skipped by a lot of users, and “leave open” by 2 users, without a close vote cast, can just be auto completed on the bases that it will hardly ever result in the question being closed.

Some more thoughts on this.

I think we have 3 “sets” of questions reviews.

  • Bad: Questions that clearly should be closed.
  • Good: Questions that should be clearly left open.
  • SoWhat: Questions that are of little value, but not so bad as to get into the “Bad” set. Do we care what the outcome is on these?

I expect that most review tasks are in the “SoWhat” set, and that most people will agree that review tasks in the “Bad” set are more important.

So if there was some way to predict if a review task was likely to be in the “bad” set, it could be moved up the review queue so it got completed quicker.

If it could be predicted that it was very unlikely that a task was in the “bad” set, then it could just be removed from the review queue. (It will go back in if it gets anther close votes.)

I expect that by looking at how long reviews take, how many skips, the past history of the reviewer etc, each review would be found to provide more information on the outcome, then is just picked up by looking at the number of “close” and “leave open” votes.

Improve this question
8
  • 1
    I'm a Computer Science Undergrad. Looks interesting to me.
    – Ranveer
    Feb 18, 2014 at 11:37
  • @Ranveer, what would it take to get you to consider it as a 3rd year project? How many AI options are you taking?
    – Ian Ringrose
    Feb 18, 2014 at 11:41
  • Currently I'm in my second year and as far as Machine Learning is concerned, I've only read some of it online. I might opt for a course in my third year.
    – Ranveer
    Feb 18, 2014 at 11:43
  • 4
    ... you mean, like this ;).
    – Matt
    Feb 18, 2014 at 11:48
  • @Matt: Looks like it, but with slightly more info. available (that contest is about preventing questions being posted I think).
    – George Duckett
    Feb 18, 2014 at 11:49
  • @Matt, the key working on that is "including everything we knew about them right before they were posted", I think the review system gives a lot more useful data that may be able to predict the outcome.
    – Ian Ringrose
    Feb 18, 2014 at 11:57
  • I would've loved to do something like this for my undergrad dissertation. I think the current API wouldn't offer enough for this to work though - how would SE open up the system enough for an undergrad to develop against?
    – Joe
    Feb 18, 2014 at 16:15
  • This is a very appealing research topic/project
    – Joel
    Mar 2, 2014 at 0:55

1 Answer 1

Reset to default
3

It could be a great help for the community. More important it would be to find the characteristics of bad questions and not allow them enter the system at first.

If you look at the question, you know when it's poor, but it's hard to say what exactly makes it poor (when you have to write a program finding such poor questions).

I'd start from Markov-Chains analysis for both closed and unclosed questions. As the first goal, detect assignment questions. The assignments are written using a bit other language than that used by developers to describe specific problem. AFAIK there are researches on using Markov Chains to detect the style of particular author (plagiarism detection).

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .