The key is that you shouldn't be voting people up to begin with. You should be voting answers (or questions) up.
The idea is to promote good posts, not good people. If people happen to write lots of good posts, then great - but judge based on the posts, not the name.
For example, when you voted several of this person's answers up, did you look at the other answers in the same questions, and vote up any other good ones? If so, that's fair - but if the only candidates for your upvotes are for one person, I think it's reasonable to object to that behaviour. (Aside from anything else, it promotes positive feedback on rep even more than the normal situation...0