Rather than thinking of reputation as being "totaled" from a list of accumulated downvotes and upvotes; you should think of it as being stored as a single value at each moment. It is updated each time a vote is cast, but it doesn't have any memory of previous votes.
I can see why you'd think it does; the "total score" model does a good job at representing how post scores work (people can even view the number of downvotes and upvotes separately for a post). But post scores actually do not have any direct effect on the reputation earned by a user. Whatever correlation exists is due to certain actions which have an effect both on the post's score, and the post's owner's reputation score.
For example, an upvote always adds one to the post score, but it adds five or ten points to the post's owner's reputation score (depending on if it's on a question or answer). A downvote always subtracts one from the post score, but it subtracts two points from the owner's reputation score, as long as this would not put the owner below one rep.
This disconnect certainly can be counterintuitive, but even if it is a problem, the system has been in place for too long for it to be feasible to fix it. Some historical failed proposals: Should we reduce rep bonus for upvotes on posts with a negative score?, Should up and down votes cancel each other?