I've noticed that I've lost some reputation when users are removed, since that's specifically called out in the inbox messages. However, I am curious if lost reputation from downvotes the user gave are also restored in that situation. All reputation lost due to deleted user accounts were evenly divisible by 5 so it doesn't appear to be the case. (i.e. you gained +4 reputation because a user was deleted).
I only ask because in a particular question I answered it appeared as if someone went through and downvoted the question and each answer for no apparent reason. Looks as if someone can damage your reputation and you can never get that back.
For me personally, I have enough reputation that the amount lost isn't that much but for someone just getting started it could be a problem.
The suggested duplicate did not cover the reputation change related to downvotes specifically so that was an open question. Based on my observations the few times I had reputation changes due to users being deleted I never had a rep change value that would suggest downvotes were being restored. Maybe they never downvoted me, or maybe the number of downvotes always ended up being a multiple of 10. It was clear about the removal of upvotes though.