While it's certainly possible some people fear downvotes because of SE reputation impact, I believe as others here appear to that the predominant reason is social in nature, largely unrelated to numerical reputation. Specifically, I believe downvotes represent a social threat.
The SCARF Model
As discussed in this paper, human beings are wired to view the world in terms of threats and rewards. We've long understood how the body reacts when being physically threatened (e.g. by a rushing tiger), but recent research has shown that social threats are experienced with the same brain circuitry and intensity as physical threats. These social threats (and corresponding rewards) can be thought of as being composed of five basic elements, as shown in the figure below. I believe each of these comes into significant play with respect to downvotes.
Status
The act of being judged is in itself status-lowering. When someone judges you or your work negatively, the effect is amplified. Further, a downvote is persistent, both in terms of the net vote totals for your post and, for those who can see it, the total number number of downvotes. As a result, you perceive your status as permanently lowered as long as the post is visible.
On MSO, the problem is particularly acute because although downvotes are allowed/encouraged for expressing disagreement, there is no way for anyone to distinguish a quality downvote from a disagreement downvote. So disagreement downvotes carry all the "baggage" of quality downvotes in addition to the fact that being disagreed with is status-lowering in and of itself.
Note that while in some extreme cases, numerical reputation impact can represent a "status" threat, it is usually minor, typically mitigated by more positively weighted upvotes and masked in one's total reputation.
Certainty
The threat of downvotes creates uncertainty because you don't know what the reaction of the community will be, particularly on MSO. While quality related factors such as completeness, correctness, etc. are largely under your control, matters of agreement or disagreement are generally not.
Autonomy
The existence of the "question ban" means that autonomy is at risk as well. The fact that the specifics of the ban are not published means that users have no idea whether a question will trigger this ban and no specific knowledge of what will be required on their part to undo the ban, adding to their uncertainty.
Relatedness
Downvoting, and particularly massive downvoting, communicates "you don't think like we do", "you don't belong here" and "you're not one of us", independent of the intent of the downvoter. This is particularly impactful when someone is new to the community and unsure whether they will be accepted.
Fairness
This attribute is second only to "status" in terms of the weight it carries. Unfortunately, the ways in which downvoting is experienced as unfair are myriad, particularly on MSO. Downvoting being anonymous, downvotes without comments (or comment upvotes), disagreement downvotes for questions that aren't bona fide feature requests and high rep users advocating "downvote whenever you feel like it" - these things all strike many as being arbitrary and/or unfair.
Conclusion
While it's true that upvotes represent a corresponding "reward system", a key research finding was that human reaction to threats is generally an order of magnitude stronger than the reaction to a corresponding level of reward. While SE has wisely taken that into account in terms of the numerical reputation system, it's not clear whether it has been taken into account in other SE decisions, particularly on MSO.
In particular, rationalizations that the threat of downvotes will incentivize better posts may make sense in the case of objective quality criteria and a culture that largely abides by that criteria (e.g. SO), but hardly seem applicable in the case of the MSO's agree/disagree voting system and culture.
