What can cause a loss of 15 reputation?
You can lose 15 rep when someone who has previously accepted your answer removes the accept. This will result in your reputation score decreasing by 15.
You won't see this in your activity though. The only way you would know is if you kept a list of all your accepted answers and then reconciled them manually.
In response to a comment to this answer, I am adding these links in for reference. Although related they don't seem to imply whether this sort of functionality is being currently being implemented or not.
You can lose 15 rep when somebody unchecked an accepted answer. This is not shown in your graph. It can also be withdrawal of votes or a rep recalc. Both are not shown in any graph.
But you can be sure that if you lost those points without any trace, you does not had them before. (The rep number can be changed due to deleted questions. You can check http://stackoverflow.com/reputation to see your actual rep and its source.)