I am planning to do some analysis on duplicated or similar question posts from Stack Overflow. Therefore, I need to collect the posts (regardless of question or answer) as the ground truth data, which are duplicate or similar to each other.

Is there a systematic way to collect such data?

I found there is CloseReasonTypeId=101 which indicates the post is duplicate to another one. However, very few posts are flagged, leading to a very small data set. Is there a better way to do so?

  • 8
    Oh how I wish there were...
    – curious
    Dec 4, 2019 at 20:44
  • You could google the titles of every question and see what other SO question comes up and then manually check them. Dec 4, 2019 at 22:26
  • 2
    Related: Academic papers using Stack Exchange data. Includes (my emphasis): 1) "Yuji Mizobuchi, Kuniharu Takayama. Two improvements to detect duplicates in Stack Overflow. 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2017)." and 2) "M. Ahasanuzzaman, M. Asaduzzaman, Chanchal K. Roy, Kevin A. Schneider: Mining duplicate questions in Stack Overflow ?. 13th International Conference on Mining Software Repositories (MSR 2016)." Dec 5, 2019 at 3:07
  • 3
    If and when you publish the result in one form or the other, please make sure to spell Stack Overflow correctly. Dec 5, 2019 at 3:09

1 Answer 1


Sure, here's an SEDE that'll pull posts closed as duplicate as well as the first dupe target:

  'https://stackoverflow.com/q/' + CAST(p.Id AS nvarchar) AS [Post],
  'https://stackoverflow.com/q/' + CAST(JSON_VALUE(ph.Text, '$.OriginalQuestionIds[0]') AS nvarchar) AS [Dupe]
FROM PostHistory ph
  JOIN Posts p ON p.Id = ph.PostId
  p.PostTypeId = 1 -- Question
  AND ph.PostHistoryTypeId = 10 -- Close event
  AND ph.Comment = 101 -- Dupe

Here's the query in action

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .