Actively downvoting bad answers can be costly, especially when you are a junior user in that SE (little reps). However if we don't penalise downvoters, we fear users might abuse this to downvote answers that aren't too bad. How about reverting the penalty when the answer's (or question's) score is less than certain limit?
Several suggestions on the non-penalty criteria (the actual implementation may be different):
- No penalty if score <= -2
- No penalty if the answer gets 3 or more downvotes, and it has a negative score
- No penalty if it gets deleted