Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

How would you go about finding duplicates questions on Stack Overflow? More generally, given access to a data-feed of the Stack Overflow material (titles, questions, answers, comments), what would be your design of a program which would find the 'most related' questions. "Most related" can be alternatively measured as 'most likely to be a duplicate' as well as 'most likely to be on the same topic'. These are co-related, but not identical.

Further, what programming language would you choose for this task? In particular, if your choice of language does not essentially boil down to 'because I know there are libraries to do most of this', then why that language?

Finally, do you think your solution would be fast enough to be used as a filter on Stack Overflow questions, to help reduce the amount of duplicate questions?

share|improve this question

migrated from Jun 18 '10 at 20:39

This question came from our site for professional and enthusiast programmers.

SO tries to do that already. When you type in your question title, it tries to find questions with similar titles. How well it works, well... that's another matter. – FrustratedWithFormsDesigner Jun 18 '10 at 20:39
Hmmm.. this probably should have stayed on SO. – Jon Seigel Jun 18 '10 at 20:45
@jon really? it's so broad and speculative as to be NARQ from my perspective. – Jeff Atwood Jun 18 '10 at 21:58
@Jeff: Possibly, but I'm pretty sure it doesn't belong on Meta. – Jon Seigel Jun 18 '10 at 22:05
I was really rather careful in how I phrased my request so that it belonged on SO rather than on meta. I did not ask for a 'better' way to find duplicates or complain about duplicates. I asked for a design for how to achieve this - a programming question. – Jacques Carette Jun 19 '10 at 1:17
If so many people are concerned about 'duplicate' questions. And so many people get involved in 'identifying' duplicate questions then why not build a mechanism to allow users to link/amalgamate question. Enough votes and the question gets amalgamated. – Chris Jun 19 '10 at 6:38
@Chris: good idea. – Jacques Carette Jun 19 '10 at 14:05

Why not download the data dump and give it a go yourself!

share|improve this answer
Because this is really really far from my area(s) of expertize. Code generators I can write. Symbolic math software, that too. Interpreters, tick. Data mining with an objective function? Natch. – Jacques Carette Jun 19 '10 at 1:20


share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .