I do not think we should be removing old answers based on score.
On the issue of noise, they don't actually add noise if you are in the 'votes' view, because there are simply at the bottom of the page, so they will only bother you if you voluntarily go to view them.
In terms of the actual removing of the answers, this is a very bad idea. We have in general no idea why this specific answer was downvoted. Starting with an example from MSO (Is "Don't Do It" a valid answer?) That question has a negative score for me. It does not mean that my answer is bad, offensive, wrong, or awfully written. It simply means that the sum of people viewing the question disagreed with me. It is important to keep these answers around, because it articulates both sides of the debate. If you delete all the MSO answers with a negative score, you'll end up with a lot of one sided debates, and no real learning.
On the SO side, I have two responses with negative votes.
Algorithms: interesting diffing algorithm
In this case, the OP (or someone tactical) downvoted almost every answer, even though I made a few edits in direct response to the author, and ended up providing code that did exactly what was required (and updating it as the OP repeatedly edited the question). I don't feel that I deserved the downvote, but that is the way it goes sometimes. However, if that were to get deleted because it has a downvote, I'd consider that offensive.
In this question as well, I was voted down twice for presenting an opinion that was contrary to other people's beliefs. Again, I don't mind the downvotes, but I maintain that my answer should be there for people to read. Even if they disagree, they will be better off if their assumptions are challenged, rather than pretending that there is no opposition.
Even 'bad' answers are educational, because they provide "patterns to avoid".