Let's face it, how many can say that never doubted to down-vote a question or answer because of the one point of reputation loss?
Besides that, it is unfair for any user to lose points because of down-voting poor questions and answers.
There is a "Critic" badge for down-voting for the first time, so I propose to create a "Judge" gold/silver badge for users who down-voted a considerable (to be defined) amount of questions/answers.