adding an anonymous comment wouldn't help much, you just would get comments like "......".
If your answer/question is correct/good it will likely be upvoted (up to the level it deserves (if not more)) resulting in an additional +8 for each downvote:
Example for an answer with a net score of +2:
what | upvotes | downvotes | NET | reputation
------------------+---------+-----------+-----+------------
without downvotes | 2 | 0 | 2 | +20
with one downvote | 3 | 1 | 2 | +28
...
assuming the system works... I've seen this happening often on Stackoverflow...
related: How do you react when someone votes down your question?