In the beginning I found it confusing as a new user that you received negative rep for downvoting someone's answer. In the reputation overview, the description said "downvoted", which is only one letter different from "downvote". I therefore suggested that the events should be renamed to "received downvote" and/or "gave downvote" or something else that is clear and appropriate, so as not to confuse any other newcomers.
You can already hover over that text for a more detailed description:
downvoted a post
post was downvoted
Additionally, when you issue a downvote, you only lose 1 reputation, whereas when you receive a downvote, you lose 2 reputation. Just going by the number at the left, you should be able to figure it out.
If all these indicators still aren't clear enough for you, you'll need a bit more evidence to convince them that this is a real issue, rather than just you being slightly confused.