Consider this scenario. Fred is scrolling along on MSO, when he comes across a question that captures his interest. He agrees with the idea that we should be able to see our own deleted questions and answers. He upvotes the question, and looks at some of the answers. He sees one that says something to the effect of
I don't like this idea, because "blah blah blah.
Fred doesn't agree with his reasoning, so he downvotes that answer. All of a sudden, he notices that his rep has gone down 1 point! He is dumbfounded, because he has always been told that votes are different on meta.
In this scenario, Fred is being essentially being punished for having an opinion. Although 1 rep point isn't much, there is still the problem that after a while, these points add up. So I am proposing that on MSO, downvotes on answers should not have a reputation effect, because votes are different here. Votes on Meta apparently indicate disagreement. So why punish the user for disagreeing?
It might be a good idea to keep the -1 on answers for questions that they have answered on, however. This would hopefully prevent the user from downvoting every other answer. Maybe we could even increase this limit.