The reputation change is per vote.
You ask a question and it gets 6 upvotes and 3 downvotes. The net change in your reputation is
6 * 5 = 30
3 * -2 = -6
+24.
There are some exceptions:
- You can't go below 1 reputation, so no matter how many downvotes a post gets, you'll never go below 1 reputation
- There is a 200-point daily reputation cap. If you get 200 reputation points in one day, any "extra" upvotes that day will count for 0 reputation
- Votes on Community Wiki posts do not award or remove reputation