I noticed on the Gaming meta, that downvoting didn't affect my reputation at all.
I suppose it has to do with the fact that Parent + Meta rep is linked, but downvoting comes with no penalty.
I'm sure this issue is already known and applies to every beta site, I just didn't see any posts about it.
Edit: I'm an idiot - the downvote in question on the parent site was community wiki. Downvote on the parent site works fine, but the fact that the meta downvote doesn't count was puzzling.