Is it fair to be downvoted when other people's answers are similar and they didn't get downvoted AND the person who ASKED the question says he didn't downvote you?
No, it's not.
However, life, Stack Overflow included, is not generally fair and you'll be happier if you don't let yourself get too hung up on that. In the case of Stack Overflow, when the stakes are just mostly meaningless numbers, it's definitely not worth getting too worked up about a downvote you don't think is fair.
People can downvote for whatever reason they like. Your best bet is to make sure that your answers are generally awesome enough that the crushing weight of upvotes makes the downvotes hard to notice.
Did the downvoted person repeat an already existing answer? Repeating an answer with your own words is only useful if you are able to repeat it in a much better way. Else the question will climb up, earn attention, people will spent time reading it and ask themselves, why you repeated an existing answer, instead of voting the similar answer up, maybe giving a comment about a minor improvement, possible.
If you show us which answers are judged so unevenly, we might try to find different explanations. Else, it is just a speculative exercise, and we might be wrong.