One issue I have with StackOverflow is that sometimes, a well-meaning Stacker will answer my question incorrectly. I don't take offense, and it seems rude to down-vote a well-thought-out answer; especially in the case that the reason that the answer is wrong contributes to the question. See here for an excellent example: http://stackoverflow.com/questions/2180306/windows-identity-foundation-how-to-get-new-security-token-in-asp-net
You can see that the poster meant to help, but didn't know that I had already tried doing what he suggested. I certainly wouldn't want to down-vote him, but I think it would be great if there were some sort of "This is not the answer" flag we could set.
"This is not the answer" will not cause anyone to lose reputation. It will simply indicate that the answer isn't helpful. It will be especially helpful when the OP comments why the answer isn't helpful. It will help Stackers give better subsequent answers and improve the quality of the discussion. Furthermore, it will go a ways toward solving the difficulty of distinguishing between answered questions that the user never bothered to accept an answer; and questions that have received an answer, but not a good answer.
With the current system, when the OP receives an unhelpful answer, all he can do is comment that it isn't helpful and down-vote if the quality is particularly low. This could be especially helpful in the event that a question receives an answer from a high-rep user and passer-bys give several up-votes because they're familiar with the user (without understanding the question and answer) and possibly put an unhelpful answer on top.
I would suggest a blue or purple (not red) icon for "This is not the answer". A red icon suggests that the poster who answered did something wrong; this is not necessarily the case!