For example a user was removed and I gained 2 reputation from them in total before. That means that I lost 2 reputation. Is there a way to find out about what sorts of voting activity was done in my account from that user?
No, you can't find out who voted for what - the problem highlighted by gnat in his comment has been address by basically not removing the votes of prolific voters.
That would make voting non-secret.
It's a big part of why you can't self delete if you've voted at all and moderators can't delete if you've voted a lot. The team have to evaluate the impact of deleting your account and take the appropriate action.