I was nicely surprised to get Unsung Hero badge much sooner than I expected. Someone downvoted my post with score 1, so it got score 0.
However, I believe this behaviour is not correct. The badge tries to encourage help to beginners and/or low traffic tags, i.e. in cases when your answer doesn't get enough attention to be rewarded by upvote. In both cases you most likely end up with an unscored answer.
Whereas if your post has got +1 and -1, it actually did got some attention, so it should not count towards the badge! The badge was not meant to encourage low quality answers, but low attention answers!
I believe only unscored (and not all zero scored) answers should count.