When downvoting an answer while reviewing, I lose
-1 rep, that's fine, but its seems like the
-1 rep comes back at some point? If that's true when does that happen?
If you downvote an answer you lose one reputation. If the answer or the corresponding question gets deleted you get your reputation back. Note that you can also undo your downvote at a later point if the answer has been edited.
So it's not a result of your review process. It's more likely a result of the collaborative review process resulting in the answer being deleted.