Recently I lost some reputation when a user was removed:
Out of curiosity, is there any way to know which user was removed?
No, there isn't.
This information shouldn't be made accessible to the public, for good reasons. When a decent number of votes are invalidated, it's either
Only the first case, which is the rarest, is info you should/might obtain, and the others are what mods and Community Managers deal with.
The other cases are most probably people you've never come across, and you wouldn't recognize them even if you knew that acoolguysomething99 voted on your posts.
Furthermore, as the comment and the other answer indicate, voting is always meant to be anonymous, and only the extreme cases of suspicious voting are revealed to mods. Telling you someone did vote on your posts would negate that.
Voting is anonymous.
If we revealed who placed those votes when the user was removed, it would no longer be anonymous.