I want to ask, how you check for duplicates.

Background: The company I work for has a wiki, Confluence, set up by me. With more and more articles coming along, duplicates are starting to pop up. They're not really duplicates, since their title and tags distinguishes them, but their content is mostly the same. We've already introduced a "search before post" rule, but it only helped a little.

Now, to prevent duplicates, I want to check for similar articles based on the title. I've found out, that you could use the Jaro-Winkler distance to get the similarity of words, but that won't do any good I think, however, I'm curious, how other wikis, Stack Exchange and other CMSs do it.

An example, why we need such a thing: Worker A searches for an article about the setup of the shipment software of DHL. He doesn't find one, so he writes one. One week later, Worker B searches for the same, but uses a slightly different terms and ends up not finding the article, because the words he used to search aren't in the title nor the tags, so he creates an article as well.

So, as said earlier, how does Stack Exchange find "question that may already have your answer"?

  • They just work like a search engine does. Not sure the question is really related to SE. Feb 1 '17 at 8:57
  • 1
    I guess asking about SEs own duplicate system is on-topic; asking for advice on how you should implement such a system and how other sites handle duplicates is certainly off-topic though.
    – Cai
    Feb 1 '17 at 9:03
  • Related / dup: meta.stackexchange.com/q/160100/245360 Feb 1 '17 at 9:08
  • Very relevant: Project Reduplication of Deduplication Has Begun!
    – Cai
    Feb 1 '17 at 9:08
  • @Cai Where should I post the question, how other sites do it? Because I don't need just one reference. The posts linked here are good though, thanks for that Feb 1 '17 at 9:18
  • Honestly, I have no idea... maybe Webmasters, you'd have to have a look at their help/on-topic pages, look through existing questions there and possibly ask on their meta though.
    – Cai
    Feb 1 '17 at 9:21
  • 4
    I think finding similar posts is just an Elasticsearch query with the words in the title matching existing titles, for example https://stackoverflow.com/search/titles?title=is+this+algorithm+&_=1485940845261
    – rene
    Feb 1 '17 at 9:25
  • 14
    Voting to leave open. Questions about how SE works "under the hood" are on-topic (although the answer may be "that's confidential"). We also have Which tools and technologies are used to build the Stack Exchange Network?
    – S.L. Barth
    Feb 1 '17 at 10:50
  • 1
    @rene I'm assuming the whole document (title, body, tags) is used in the search, with a more_like_this.
    – OrangeDog
    Nov 19 '19 at 13:36
  • How does it search? "Badly." Nov 19 '19 at 16:43

You must log in to answer this question.

Browse other questions tagged .