Check if a user has an identical open question on another site before allowing them to post [duplicate]

Question

There are quite a few sites in the network where content can sometimes overlap. That is, there are questions that could be appropriate on multiple sites in the network. Combine this with the fact that some askers are very eager to get their questions answered, and you can end up with scenarios like this:

• OP asks question on ELL (34 min ago)

• OP asks same exact question on ELU (32 min ago)

• Someone answers the ELU question (28 min ago)

• Someone answers the ELL question (23 min ago)

• A second person answers the ELL question (21 min ago)

• A moderator on either site (let's say ELL) is tipped off and realizes that an identical question is open and answered on both sites. (19 min ago)

• In the meantime, a second person answers the ELU question (18 min ago)

• Said moderator contacts an ELU mod to determine which site the question should be migrated to for merging. Usually the answer is "the first one it was posted on", but sometimes the question is more appropriate for one site than the other. (17 min ago)

• Moderators agree on destination; duplicate question is migrated to ELL (15 min ago)

• Duplicate is closed and merged so that all the answers appear in one place (13 min ago)

• Users who took the time to answer the question on either site realize that answers had already been posted on the other site and that they quite possibly wasted their time answering in the interim. If the question was migrated away from their site, users realize they have lost rep for that post. (7 min ago)

• It is determined that a whole lot of time and effort has been wasted for everybody (58 seconds ago)

The solution? When a user attempts to post a question, check if they have an identical question open on another site. If they do, do not allow them to post it.

This would save a lot of time and effort for everyone involved. Cross-posting identical questions between sites is not allowed, and having a feature that disables it would save time on the part of both answerers and moderators. I simply used ELL and ELU as an example because I'm familiar with it; I have a feeling this is an issue on other sites as well, or else the answer about disallowing cross-posting would never have been necessary.

Note that I am not asking that this check include closed questions; if a question has been closed as off topic on one site, it's perfectly acceptable to try and post it on a site where it might be on topic. But if a question is open on one site, it shouldn't also be open on another. We want all the answers in one place and to not waste answerers' time repeating things.

This can get especially hairy because unlike most things moderators deal with, it cannot be accomplished by yourself. You must get in contact with a moderator from the other site—andthat doesn't always take just a few minutes. While you're waiting to discuss the final destination of the question (or if you know it belongs on your site, but can't take care of that because you don't have that migration power) more answers will accumulate on the questions, thus wasting more answerers' time.

Answer 1

It's a fair idea, but it would not be feasible to implement.

What's the first reaction of a user going to be when they get a message like You already have an identical question on xyz.stackexchange.com? Right, they alter it slightly so it passes the filter.

The moment when it's no longer about a simple 1:1 comparison, you're in real hot water programming-wise. You have to define question similarity, and think about where you set the threshold. How similar must the contents be so the question gets blocked again? What if the algorithm recognizes a question as "identical" that isn't really, but uses similar words? What if the user opens a Thesaurus and simply start using different words that the similarity detection cannot reasonably be aware of? And so on and so on.

Answer 2

How about instead of preventing the user from posting their question again, we merely detect that the posting is duplicate and point them towards a better solution via an appropriate dialog box.

In my experience (of ELL and ELU), the reason for the duplication of the question is typically because some user posts a question on ELU, is told by users of that site that the question is a better fit for ELL, and they then duplicate the post on ELL. This then violates our understanding with ELU that duplicated questions are always closed on the second site they are posted to, so we immediately close the question on ELL, leaving the user confused as to what to do next.

Worse - if answers were posted on both sites, we end up with complicated shenanigans in merging the two posts, and possibly also migrating the ELU one to ELL once all of the dust settles.

This is a particularly unsettling and confusing for users that don't know the rules. They came to ELU to ask a question, got told to post it on ELL, and now all of a sudden their new question got closed on ELL, there's a whole load of angry people running around shuffling their question between sites, and all the while they just want an answer to their question; they never meant to break the rules or be a pain.

So instead of preventing the question being posted (which will add extra bafflement to the user who is only following the request of the users of the first site to move their post, but simply doing it in the wrong way), we simply tell them that what they are doing is the wrong way to solve their problem at the point where they are doing it wrong - and more importantly, tell them the correct way to migrate their question to the other site.

It is easy for the software to detect identical questions being posted on two different sites. When it happens and the users are linked, pop-up a dialog that says something along the lines of:

Oops! It looks like you're posting a question to ELL that you've already asked on ELU! Did you mean to migrate it instead?

And if the question is by a different (i.e. not linked user), then perhaps pop up a box saying:

Oh! It looks like there might be an answer to that question on blah.stackexchange.com. Would you like to review it?

In both cases, the intention here is not to stop "malicious behavior", because I don't think that is where these duplicated questions are coming from. Instead, the intention is to help users before they accidentally do something that is both counter to Stack-Exchange's rules, but also counterproductive to their own end of just getting an answer to their question.

Malicious users and users that click past the dialog with a cursory "out with ye, foolish computer! I know what I am doing!" will perhaps need manual intervention to close/migrate the question. But that's still much less effort for moderators, and much less confusing for the user, than the status quo of this happening frequently for the default case of non-malicious users who don't understand the mechanics or rules of Stack Exchange who are just trying to get an answer to their question, and then following the advice of the users on that site.

---

Technically, this isn't as hard to achieve as you'd think, given the constraints of the StackExchange network. Here's how I'd solve it:

1. Create a single database for the entire network of (QuestionTitle, LinkedUserId, Site).

2. Every new question gets a record added to the shared database. The load against this machine is not horrendous because the amount of information is relatively low and the number of updates/requests is proportional to questions asked across the whole network which is only a very small part of the overall traffic to the StackExchange network.

3. Now telling if a question has been asked before is a single query on the database: "SELECT * from db.questionTitles WHERE QuestionTitle = %1". This also tells you if the question was asked by the same user or by a different one, and which site owned the original.

It's also not necessary for us to pre-populate the new server with all of the historical data of the entire Stack Exchange network. The cost of a false negative is low (i.e. we don't pop up a box telling someone on ELL that they asked that question six months ago on ELU). The primary benefit of the new database is in preventing time-local copies of the question being asked. Adding the full question/user/site history of the entire Stack Exchange Network would therefore be a nice to have rather than a need to have feature, if it's likely that doing so would be prohibitive.