Are duplicates creating broken windows?

Question

I think the Eternal September is dooming Stack Overflow to have the same questions asked over and over again, and the people who don't want questions closed are winning.

Why don't people want questions closed? Because then those questions are a candidate for deletion? Why don't people want questions deleted? Because then they lose reputation with easily asked questions that are popular.

As an example, this question was asked today. It's already been viewed 208 times, and the question asker has 16 upvotes for his question. He has a fair amount of reputation, so it isn't as if he's new.

Yet, that same question has been asked at least half a dozen times:

• Why is using a View bad with SELECT *
• Which is faster or best, Select * or Select Column1, Column2, Column3
• Performance issue in using SELECT *
• Is there a difference between Select * and Select [List each Column]
• Select * Except

Another user has suggested getting rid of the close button entirely. I don't agree with that, but evidence suggests that we're not doing enough to stem the tide of highly duplicated questions (duplicated 4 or more times).

Problems

Problem 1: Users can game the system by asking highly duplicated questions and receiving reputation for them.

Net effects:

• User keeps reputation
• User is rewarded for asking a duplicate
• This user (and other users) have further incentive to do it again.

Problem 2: Questions get duplicated; content is splintered throughout the system.

Net effects:

• Broken windows. No upkeep makes this like every other venue for Q & A out there.
• "Expert Users" could start to leave as soon as another system comes out that gets this 'right'. It's been a problem since the Usenet days, but it's one that should be fixed.
• Stack Overflow isn't living up to its core mission of having an authoritative source for each question.

Problem 3: Users that care about the system and want to keep it as an authoritative source get discouraged by the deluge of duplicate and the seeming lack of moderators stepping in.

Net effect:

• Stack Overflow isn't maintained as well by the community that cares about it as something more than a better version of Reddit or Hacker News.

Solution:

• Stop rewarding reputation for questions that are closed as duplicates. Revoke any reputation granted for those questions (This is a draconian solution, I know).
• Implement the features I've suggested here. (Note that those features have been suggested in other places by other people as well.

Examples of questions with 4 or more source duplicates:

• Should I learn Python 2 or 3? (1,2,3,4,5,6,7)
• Password Encryption in C#
• Should I master one language or learn multiple languages?
• Free Continuous Integration Solutions for .NET(1,2,3,4,5,6,7,8,9,10,11,12)
• Stored Procedures vs Inline SQL (1,2,3)
• How do Married Programmers find time to work? (1,2,3,4)
• Protect C# code from reverse engineering (1,2,3,4,5,6,7,8,9,10)
• When should I use do-while instead of while loops? (1,2,3,4,5,6)
• Automated Deployment in ASP.NET? (1,2,3,4,5,6,7,8)
• Best Resources for learning Objective-C (1,2,3,4,5,6)
• Subversion Repository Organization

Answer 1

The answer to a lot of your points, as has been brought up before, is better merging.

The sooner the posts are merged, the better, and we can just leave a locked stub to help the search engine out later.

Answer 2

Ironically, isn't this question itself a duplicate, thus illustrating why these kinds of unintentional duplicates aren't evil ... or even wrong?

What is with people who answer questions that are known to be dupes?

I agree they should be closed, but I'm having a hard time understanding why people are so upset that others get a trickle of rep for duplicates asked and answered in good faith.

The only time I get irritated with duplicates is when the asker or answerer knows about the duplicate and proceeds anyway, and is thus asking or answering to game the system but ... clearly you didn't know about the duplicate when you asked this question, right?

I just think it's amusing that everything you're complaining about w/r/t duplicates applies to the very post you created. Maybe the next time you check for broken windows, start with your own home?

Answer 3

+1 to removing duplicates and revising SO for quality

The problem with duplicates is, they get in the way of new questions that haven't been answered yet.

I'm frustrated with SO because I spend 5 hours searching through duplicates looking for that one edge-case answer that will resolve all my problems. Then, when I've exhausted all possibilities, I create a question and the only answer I get is something along the lines of "that's interesting, why'd you wanna do that." And worse, I don't have the rep to down-vote the answer as subjective. ::sigh::

What happened to the dream of 'a site to that contains canonical answers about tech questions.' I really liked that dream.

I've copied my answer from The bike shed problem and SO as an example of duplicate questions that look suspiciously like low-hanging-fruit that I kept seeing over and over while I was searching for an answer.

Here's a frob this widget question from me that is both, not a paint shed question, and something that I really need help to answer.

The problem is, there are 400 questions something along the lines of.

Importing best practices

• What are good rules of thumb for python imports?
• Should Python import statements always be at the top of a module?
• python import coding style
• Python - when is ‘import’ required?
• what is philosophy of using import in Python?
• Python importing

Still best practices - importing in the middle of a file

• good or bad practice in python: import in the middle of a file
• local import statements in python
• Is it pythonic to import inside functions?
• Does python import all the listed libraries?

*How are 'import module' and 'from module import '

*Or, why is 'from module import ' bad

• In Python, what exactly does “import *” import?
• ‘import module’ or ‘from module import’
• from X import a versus import X; X.a
• Properly Importing Modules in Python
• In python, why is “import *” bad?
• Python: How can I import all variables?
• What’s the purpose of “import package”?

Pertaining to cyclic imports

• Python: Cyclic imports
• detecting circular imports
• Python initialization and circular reference counts
• Should I worry about circular references in Python?
• Circular dependency in Python
• Do all dynamic languages have the circular import issue?
• Circular import dependency in Python

These are the 5 base cases, now mix and match every permutation of these questions until you get 400 unique questions and you now have 400 bikesheds sitting between the highly experienced/talented SO users and my hard/impossible question to answer.

Whatever happened to the concept of SO containing canonical answers to programming related questions. In this case SO needs 'A canonical guide to using python imports' thread so they don't block out the really difficult edge cases like mine.

The only thing I've seen that's canonical on SO is threads about programming:

• cartoons
• jokes
• quotes
• wtfs

If the community wiki mode was supposed to kill the rep-whoring of useless questions, why are those questions still dominating SO?

Edit: So 400 is an obvious exaggeration. So, to demonstrate I added links. Welcome to bikeshed-land. If you don't see a common theme I'll give you a hint. It rhymes with rubjective. We're talking about low hanging fruit here and a lot, if not most of those question have 5+ upvotes.

Update: It appears as though this issue has been addressed. Duplicates aren't bad, they're just different ways of asking the same question and should be marked as duplicate and linked to where the answer is addressed. In other words, this answer doesn't really apply anymore.

Answer 4

This relates to another point I've brought up in the past regarding how valuable duplicates are and if we should bother to keep them around. The stated purpose of allowing duplicates to linger is that it helps cover a wider range of search terms. But this purpose is somewhat moot if A) people aren't bothering to search first and B) they find the duplicates but are compelled to reask the question anyway because it's easy Rep and a greater part of the community seems to be (wrongly) OK with it.

In short, there doesn't seem to be anything in place that really discourages asking of duplicate questions, meaning that duplicates will likely keep uselessly piling up. As far as a solution, maybe having unlimited close votes for just the "duplicate" reason (while still limiting reopen votes to the daily cap) will help to keep duplicates closed or close them faster. I for one run out of close votes rather quickly, so I often have to just leave comments on duplicates I find.

Answer 5

I think it would be interesting to do a bit of analysis to see if your interpretation of people's motives is true: i.e.,

• is it low or high rep users asking duplicates?
• are some guilty of repeatedly asking duplicate questions?

My own feeling is that duplicates have a lot LESS to do with gaming the system, and a lot MORE to do with friction. It's just easier to ask a new question than to find an existing one. Some of that friction is unavoidable (we're all lazy, and most of us don't know how to read)..

The obvious pro-active solution is to make search better. The SO search box is not great, when it's not totally useless (try the difference between looking for "F# books" and "f# books"). You can opt to search via Google, but again, that's just more friction.

@Jeff, it could be interesting to try and analyse the log files to see how many users actually search and/or click on proposed duplicates before asking questions.

The "post-active" solution is to make clearing up duplicates easier, which is where the question Ether linked to was going. I have also previously suggested added incentives for pruning one's own duplicates.

More carrot, less stick.

Answer 6

Minor Irk. Some of those examples aren't duplicates. Why is using a View bad with SELECT * is talking about views. Select * Except is talking about selecting everything except a column so shouldn't be closed as an exact duplicate (even if they have similar answers).

---

Users can game the system by asking highly duplicated questions and receiving reputation for them.

I haven't found (and you haven't provided) any example of this. I don't think it is intentional. The issue is users aren't closing soon enough. From the views on some of the examples it should have been closed far sooner.

Making closing as a duplicate easier would help with the issue a bit. It is a pain to close as duplicate at the moment. (Copy link paste link, wait for box to come up, select question etc)

I would suggest that we make possible duplicates more prominent. Maybe put the list of possible duplicates above the question and make it far easier for users to agree with the duplicate. For example just a (undo able) button to agree with the suggested duplicate (Next to the list above the question).

Another idea to complement the above might be to have a "duplicate" button next to all the "Related" posts.

---

Why don't want people want questions closed? Because then those questions are a candidate for deletion? Why don't people want questions deleted? Because then they lose Reputation with easily asked questions that are popular.

This has never occured to me, but to be honest a lot of questions are getting deleted that shouldn't be. I don't know the exact statistics but I imagine a lot of the hits this site gets comes from search engines. Deleting duplicated questions that aren't exact duplicates removes alternative ways of asking something from the search engine index.

---

Stop rewarding reputation for questions that are closed as duplicates. Revoke any reputation granted for those questions (This is a draconian solution, I know).

This doesn't help the fact that the questions aren't getting closed soon enough.

If it got closed within 3 or 4 minutes, it wouldn't get very many upvotes as it would fall off the main page fairly fast. Problem solved.

Answer 7

Maybe this proposal should be its own question, but what about warning question askers if their question is very similar to a question that gets a lot of duplicates? It might sound impractical, but there's already metrics for question similarity, and it's easy enough to calculate how many duplicates an existing question has.

Answer 8

How would you want this to work as a new user, who is looking at this site for the first time.

We should look at this more like the internet. Many destinations, many start points and many ways to get there.

My use case would go some thing like this

Ask a Question.
Q: How do I ask a Question on Stack-exchange.

What should happen at this point.

The first thing a user should see is the input box to ask a question. As they type a list of answered questions is generated as it currently does. They can select which answers, answered their questions. If none of the question or answers that came up solved there problem then posting the question to the community should be the next step. I understand that you wanted to make it easy to ask questions, and you have done that excellently, may be the problem now is, making it easy to find answers.

I think we have confused the way thing currently work with the way we want them too work. Do we need to have a separate function or option for searching and asking a question. I would just combine this into one option. Ask a (question / search). Trying to keep things simple would go a long way to limiting the number of duplicate questions that keep getting asked.

Next keep track of the question as part of the users session. Now if they find an answer or question that works for them they should click the tick or up vote just like community members. Except the reputation should be handled differently for strangers as apposed to community members. Now the session which saved the question can be used to link their question to the questions and answers they find. You can start to build a meta index for questions and answers. Which should be used to improve search results.

Many people would ask the same question in many different ways. I don't think closing some thing as a duplicate, which is a high reputation problem needing 3000 rep to vote to close. I think it is a linking problem. Give people an easy intuitive way of linking similar questions to similar answers. If there are more then one way of asking a question, why limit it, keep it and use it.

It may seam like people are being stupid, asking the same questions in different ways. I don't feel it is their stupidity I think it is the interface has hidden the answers they are looking for. Maybe all they are looking for is clarity, but with out enough reputation, you cant comment on some one else's question, or answers. How can the find clarity other then asking the same question with may be different word order to emphasize the issue they are having.

Lastly it should be a lot easier to post ancillary question or comments when people need clarification. May be an "I don't understand" button which lets them ask a question on a question. This should to push the question back on to the hot question list. To expose it the community as needing clarification.

Answer 9

Just a note:

I've just started a new question with "Obsfuscating C# code" as the title and none of the potential duplicates showed up on the "Related Questions" list. The algorithm seems to have picked up on the word "code" rather than "obsfuscating" - which is understandable given that it's misspelt. Typing "Obfuscating C# code" does produce a list of questions about obfuscation.

So in this case we can perhaps let the OP off asking a duplicate.

However, it does raise a couple of points about the related questions search.

1. Should there be a spell check on the title regardless of the users settings? If the title is correct then the search can work. This could be done behind the scenes in the actual related search code.

2. Should the related questions search be run once the body of the text has been entered and use the body text in the search as well? Would this increase the likelihood of a duplicate being found and spotted by the questioner?

Answer 10

I was thinking duplicate questions could be forwarded instead removed or closed.

Forwarding questions would leave the different wordings of similar or same questions indexed & searchable while keeping the answers all maintained under the original question.

That way if someone asked "[php] How do I open a connection to a MySQL server?", "[php] How do I use mysql_connect?" is already answered, I could "answer" the former with a "forward" to the latter. If my forward became the accepted answer, or got the most votes after some period of time, subsequent hits to the forwarded question's URL would bring searchers to the one that was already answered.

Sorta like on wikipedia, if I go to http://en.wikipedia.org/wiki/Tip_ring_sleeve , wikipedia will forward me to http://en.wikipedia.org/wiki/TRS_connector

Why scrap Tip_ring_sleeve when other wikipedia users might not know it's often abbreviated to TRS? Why stiff stackoverflow users and subsequent searchers/users who don't know the right question to be asking?

Answer 11

Provided that the duplicates are well linked to each other, duplicates are, in my opinion, a Good Thing(tm). They increase the overall search space for users who are trying to find an answer to their question, because they increase the total volume of word combinations that a questioner might search against, increasing the likeliness of a hit.

Closing duplicates fine. Deleting duplicates is a Bad Thing(tm).

Worrying about them incessantly is... well, it's probably bad for your ulcer.

Answer 12

I find this thread rather amusing since my own feeble attempt at getting the SO crowd to start thinking about solving its own problem of duplication got moved to meta. I tried rather hard to make the question a programming one rather than a value judgement, but was very quickly judged to be NARQ.

Perhaps if there were much better tools for finding duplicates, and then tools for merging or cross-linking (or ...), the total level of duplication would go down?

I am absolutely sure that the reputation system can be used to reward those who find duplicates and also provide a (serious!) disincentive to those who ask questions that have already been asked before. Along the same vein, I believe that there should be a higher cost to voting to close a question - something like 20 reputation points. It certainly should not be free, it should be more expensive than a downvote.

Answer 13

Personally, I hate that we close duplicate questions because they are helpful.

First, the OP often doesn't understand why his question is a duplicate even if it is pointed out to him especially when he already read that question and it didn't help him. Closing the question only drives the people asking questions away. Sometimes, someone else rephrasing the answer in a new question changes it just enough that the lightbulb comes on. We are often missing those lightbulb moments where a different explantion might suddenly make the person understand.

Next, the one of the best ways for junior professionals to get the knowledge they need to understand their own technologies better is to answer questions. If we close all the easy ones too fast, these people don't get the chance to learn by looking for the answers for other people. This is a great deal of how I learned a lot of what I know about SQL server since I was, at the time I was learning, in a job with no peers in my specialty. We are taking that experience away from the people who want to get better, but who are truly juniors right now.

Next, the answers change over time as the technology changes. By closing duplicates too fast, we may be encouraging people to use outdated methods.

Further as new people come onto the site, they may have a better answer that the orginal question had, but if the new question is closed too soon, they are less likely to provide that answer. Most people, I think, are less likely to provide a new answer to a question answered a while back than they might be to a new question.

By closing questions too fast, I think we are becoming a site more about the elitism of the advanced user and less about actually helping people who need help.