Is there a way to systematically find duplicate or similar posts on Stack Overflow?

Question

I am planning to do some analysis on duplicated or similar question posts from Stack Overflow. Therefore, I need to collect the posts (regardless of question or answer) as the ground truth data, which are duplicate or similar to each other.

Is there a systematic way to collect such data?

I found there is CloseReasonTypeId=101 which indicates the post is duplicate to another one. However, very few posts are flagged, leading to a very small data set. Is there a better way to do so?

Answer 1

Sure, here's an SEDE that'll pull posts closed as duplicate as well as the first dupe target:

SELECT TOP 10
  'https://stackoverflow.com/q/' + CAST(p.Id AS nvarchar) AS [Post],
  'https://stackoverflow.com/q/' + CAST(JSON_VALUE(ph.Text, '$.OriginalQuestionIds[0]') AS nvarchar) AS [Dupe]
FROM PostHistory ph
  JOIN Posts p ON p.Id = ph.PostId
WHERE 
  p.PostTypeId = 1 -- Question
  AND ph.PostHistoryTypeId = 10 -- Close event
  AND ph.Comment = 101 -- Dupe

Here's the query in action