As far as I can tell downvoting is mostly useless.
I strongly disagree. As a user of the site, it's one of the strongest tools that I have available to me to improve the quality of the site.
For questions the only people who get downvoted are lower rep users, or incredibly stupid questions
This is false. There are plenty of higher rep users who make mistakes and post incorrect answers, getting downvotes, still haven't learned how to ask a good question (despite being good at answering them) etc. Now, most questions in general are asked by low rep users, so the fact that most downvoted questions are also asked by low rep users is not surprising.
downvoting doesn't have too much affect on your rep anyway
True, but the effect is more emotional than that. When someone has a question/answer downvoted they don't ignore it because the rep change is small, in most cases.
a negative comment is more useful and more of a push too change or delete the question
I've found both to be useful. I've had posts in which there were a few upvotes and zero or one downvotes to which there was a comment indicating serious problems. In such cases I've seen the authors acknowledge the problem and just ignore it, because "who cares, it's upvoted". As soon as the post gets a number of downvotes, all of a sudden there's a motivation to fix the post.
Conversely, a post with downvotes in which the author genuinely can't determine the problem, can't be fixed, and requires a comment, but the motivation to actually go and make the fix does often come from a downvote.
people commonly competitively downvote (downvoting another answer to a question that you also answered) obviously not a good behavior
I personally don't see this a lot. Certainly not enough to result in incorrect answers regularly appearing before correct answers in vote sorting as a result.
downvoting (and upvoting) are contagious, meaning one downvote leads to a bunch of others
So why should dowvnotes be removed as a result? I most certainly stipulate to that point. There will be times where a post gets an upvote or two just for being first, and then gets a few more because it's the only upvoted answer. If someone notices a problem with it and posts a comment explaining that, downvotes are the means for future readers to correct the mistake and push the post back down. This is very valuable as a feature.
even though downvotes don't effect rep that much they create a negative environment, no one wants their post downvoted, on the other hand criticizing comments, I think, are far more welcome
I tend to leave a lot of such comments, and I can tell you that it's very rare for them to be welcome. I work very hard to make sure my comments are polite, constructive, helpful, etc. even when pointing out a problem with a post (granted, I don't always succeed, but I do try). I am almost always met with defensiveness or aggressive behavior, and not uncommonly open hostility. It's much less likely for the OP to learn or improve their answer in such cases.
Other purposes downvotes serve:
They feed the question/answer ban. This is actually quite important on sites like SO to ensure that experts aren't driven away by a small percentage of users creating a large amount of undesirable content. Without downvotes it is a lot harder to create good metrics for such bans.
They provide a very clear and easy to digest summary of the communities opinion of an answer at a glance. In a world with no downvotes a post with a score of +10 and 20 comments could be one that's great and had lots of people praising it, or it could be a controversial post that 40 experts thought was terrible and 10 others thought was great. If you don't have the time to red through all of the comments explaining the problems with the post, the reader loses that information.
