Why does someone else breaking the rules and being removed cause me to lose reputation for votes they had cast on my questions or answers?
Not all of a deleted user's votes are thrown away; see this answer by Shog9 for details about how this works.
But note that allowing a deleted user's votes to stay, opens a window to abuse.
A user could come back with a new account, and vote for the same posts again. Thus giving you more points than you should have.
(In fact, I know of one user who was active on multiple SE sites, and asked to be deleted from one of them. The request was honoured, but the user later did return to that site.)
Even if a returning user is acting in good faith, they may have forgotten that they voted on your posts before. A returning user is likely to be active in the same tags, so is likely to see your posts again.