Considering the number of increasing down-votes across all SE sites (and especially StackOverflow), it really makes me wonder how many users who just "casually" down-vote other's answers know this fact that it takes down two points from their own account.
If you consider my own example, I usually never down-vote unless there is something totally wrong with the concerned answer (though I've had others down-vote my own answers for no apparent reason). But since I came to know this fact a few days ago, I've stopped doing that too. Why should I down-vote to improvise someone else's situation, if it affects my own reputation too? Isn't that a fair question?
I think we should advertise this fact more and more that down-voting actually costs the caster too. We will surely see a reduction in those "casual" down-votes cast just because someone had a fight with spouse at the morning breakfast.
Update
Okay, I did not know that the single negative point to the caster is only applicable for answers, not questions. Otherwise, I would not have asked this question in the first place, in a forum that allows the whims of anonymous users to overwhelm a question with downvotes. Had downvoting this question cost you a point, I bet you wouldn't be doing that.