You can add a comment that you did not downvote. 

And if you feel you got serial downvoted then wait a day. If it does not get reversed automatically then flag one of the downvoted posts *for moderator attention*. They can handle such cases.

In the end 2 downvotes are 4 reputation points. Like others commented: May may also just let it be.