Your downvote only accounts for 1 out of AT LEAST 5 necessary.
That means that 4 others would have to also downvote it, making the power of your own downvote weaker. And if that question ever gets any up votes (we are all well aware of people giving out sympathy votes for downvoted material), it means you need yet another downvote to offset it to keep it towards the -5 range.
Edit I should also note that getting that many upvotes on a question that could potentially hit -5 is pretty hard (it was my answer to a really bad question which led to the badge).
The biggest difficulty is in getting 20 people to spend an upvote on an answer to an otherwise terrible question. Usually once a question hits a certain amount of down votes, people will just skip over it and pay no attention to the answers inside.