Duplication is a Hard Problem, and not one that is going to go away by sheer force of will. We can all bloviate all we like about how bad the situation is but what's lacking is a comprehensive solution. So far all we have are scattered suggestions that attack tiny fragments of the problem and may have unintended negative side-effects.
I've noted in the past that the core team often has dissenting views, and there is no clearer evidence of this than on the issue of mass duplication:
Joel takes an optimistic and idealistic approach, believing that making the internet a better place is a venerable goal and that communities should have an intrinsic motivation to do it. It works for Wikipedia, right?
Jeff is more of a populist, stating essentially that given the right tools, the people will eventually do the right thing, at least most of the time, and user experience is king (no penalties for asking/answering duplicates).
Robert, whom you might be less familiar with unless you participate on other Stack Exchanges, is keener on crowd psychology and social proof. If you follow his blog posts you'll see that his message is that new members of a community are heavily influenced by the behaviour of the existing community, and that quality standards must be established early on and periodically reinforced in order to avoid an Eternal September situation. How duplicates are treated factors heavily into a community's quality standard in my opinion.
This is more than just an academic exercise or a criticism of the team. The point is that all of these viewpoints are correct - for a certain segment of the community.
Some people really do want to make the internet - or at least their community - a better place. But others are either chronic helpers or reputation hoarders and are unable to see the bigger picture.
Crowdsourcing is an effective tool - as long as the goal is strictly informational and personal motives are abstracted away.
And most people do aspire to live up to the community's standard, but there will always be a certain number of leeches and help vampires (which contribute heavily toward duplication).
It's easy to solve problems for the general case. It's the long-term accumulation of edge cases that erode communities. What we need to do, but have not even come close to doing, is gating off the edge cases without damaging the mainstream cases. It's abundantly clear that Jeff will never accept a solution that discourages participation, and it's hard to argue against his logic, considering that its precisely that participation that made Stack Overflow so attractive/useful in the first place.
As infuriating as it is for many of us to see other people piling on to obvious octuplicate questions for easy rep, levying penalties on them is not the answer. What's needed is a way to provide an incentive for minimizing answer sprawl that is stronger than the incentive to add to it, without lessening the incentive to answer in the first place.
It's usually at this point that everybody jumps headlong into a discussion of methods, without actually stopping to think about what the incentives are. There are several!
- Getting the problem solved (asker);
- Helping the asker get his problem solved (answerers);
- Teaching the asker how to solve his own problems (answerers);
- Earning reputation (answerers);
- Keeping the site free of noise and clutter (moderators, closers);
- Helping future readers to solve similar problems (closers, editors);
- Improving the overall quality of content (editors).
Probably the most interesting thing about incentives is that the asker almost universally has only one incentive, and that is to solve his immediate problem. Sometimes people may post "learning" questions but those are generally not the people who post low-quality or duplicate questions.
So the first question we should be asking is: Could we help the asker get his problem solved faster and more easily by never posting the question at all?
It's been stated that the in-site search needs to be improved, and it does - but that only helps the people who are willing to search. Primarily we're concerned with the people who don't search - those proverbial help vampires. Can we provide an incentive for them?
If you're anything like me then you've succumbed to the rubber ducking phenomenon at least once or twice. Nothing is more likely to get somebody to abandon their half-written question than having their problem solved while they're writing it. Can we do that?
We have the "related questions" section that pops up under the title box. It's actually not too bad at finding duplicates. But it suffers from a fundamental problem: People have to click on those links and navigate to different pages. They have to interrupt their work flow to investigate possibilities that only might be useful. For non-power-users who don't know how to instantly fire open all those links in different tabs, that's a major disincentive. Easier and faster to just fire off the question, right?
A more powerful system could stop these questions mid-stream. Refine the duplicate matches while the post is being written, and if any start to look "exact" enough (based on some yet-to-be-discussed scoring mechanism) then display that question and answer inline. If there's an issue of screen real estate then display a highly-visible indicator that a strong match was found and provide some AJAX-ified ability to display the content without ever leaving the current question.
How many times have you started asking a question verbally and had somebody cut you off in mid-sentence with "I know what you're going to say, and the answer is..."? I've had it done to me and done it myself plenty of times. This is the equivalent of that, except less intrusive.
Below is a conceptual example of what I'm talking about. I suppose the tooltip preview would be optional, but would be a major help here. For space reasons it would only show the top (accepted or highest voted) answer and might have to cut off the answer and/or question if it were too long.
That's one major incentive taken care of, but let's say it fails. We now need to take into account the answerer's desire to help as well as his desire to earn reputation - both very powerful incentives on their own - and try to instill an even greater desire to improve the community. To "make the internet a better place", as it were.
This is not going to be easy using any technical means. For one thing, it's not even possible to measure the success or failure of the latter objective, whereas the former objectives are easy (upvotes + accepted). I propose an alternative: Allow people to do both at the same time.
Closing the question as a duplicate does not contribute as much toward helping the asker solve his problem as actually answering the question. But could it? Quoting from Jeff:
Put yourself in their shoes. Instead of finding …
They have to deal with finding:
[closed as duplicate of Question] click here to see answers
Now, what other site requires users to do some sort of weird scroll-down, click-here-first to see the answer nonsense on the search results before they will reveal the answer? Oh yes, our old hyphenated pals. Do we really want our site to work like theirs?
And of course the answer is no, we don't. But so far, I don't think anybody's pointed out that those aren't the only options! What about this instead?
I can almost already hear some snarky person saying "hah, that's just merging, we already have that! Just auto-merge them!" Except it's not:
- The asker's question still exists in its original form and "owns" its own answers;
- Merging might require additional cleanup, i.e. removal of duplicated answers;
- If it turns out not to be as exact a duplicate as people thought, the action can be undone;
- It helps moderators immensely in determining whether or not the question is actually a viable candidate for merging (if not, then is it really a dupe?)
- It ranks duplicated answers at the bottom regardless of score, at least until an actual merge is done. So authors of duplicate answers get to keep their rep, but once a question has been closed as a dupe, they'll no longer be "featured" on top until a true merge and the original answer authors will get most of the credit. (I'm assuming that the "merged view" would support upvoting and editing of original answers as well).
So unless I've overlooked something huge, we've now wrapped up the answering incentive problem as well, without providing any disincentives whatsoever to askers or answerers. All that's left is an incentive to get people to edit the question into canonical form.
Using the above UI, we already have a head start - the ability to compare the duplicate and original side-by-side and quickly see which parts would need to be generalized. The ability to actually vote to merge the questions would, IMO, make for an excellent 20k privilege (or maybe even higher).
The idea would be that members with editing privileges would edit one of the questions (probably the original) until it completely encompasses the duplicate question, then vote to merge. The knee-jerk "oh my god, we can't let anybody but moderators do that!" reaction could probably be softened a lot by actually keeping a record of the merge and providing the ability to undo the merge if necessary (even if only a moderator can perform the undo).
What I particularly like about this is that not only does it make it easier to merge, it gives the people who contributed good answers a strong incentive to participate in the merging process. Why? Because once it's merged, then their great answers will get ranked normally instead of below all the originals, attract a bunch of upvotes and earn the authors more reputation.
And guess what other problem this solves? For heavily-duplicated questions, it solves the barrier to entry problem that several people have complained about. That's because after five or six merges, there will be so many edits to the question/answers that they'll naturally get bumped into Community Wiki mode, and (almost) anyone will be able to edit them then.
So - did I miss anything?