Downvotes (either giving or receiving) no longer affect Epic/Legendary badge progress (this changed in 2013, well after the question was asked and the other answers were posted), so this isn't really a problem any more.
There are still some residual issues, mostly related to the user interface. For example, the badge tracker appears to display downvotes even though the badge itself doesn't. Additionally, if you lose reputation due to a deleted post or deleted user, this causes the reputation loss to be charged against the day when you originally gained the reputation, but the UI displays the loss as having happened against the day when the deletion happened (source); this is probably capable of affecting the badge, but not in the way that the UI suggests.
Note that the UI issues are still a real problem! In my case, the combination of Epic together with the fact that removed posts appear to reduce the reputation cap below 200 ended up discouraging me from attempting to salvage posts by editing them (an edit gives you +2 reputation, but there's a high chance that the post will get deleted and give you a -2 on a random future day; this is awkward as it ends up locking your reputation cap at 198, meaning it looks like you aren't rep-capping properly). This is less of a problem once you have enough reputation to unilaterally edit, but nonetheless we probably don't want to make reputation increases appear to be a punishment rather than a reward.