My votes are on 0 - was I voted up and then down?
As Michael says, you got one up and one down on that answer, but you also got one down on your question: http://stackoverflow.com/questions/3288494, which is why you went -4 totally in the last few days.
If you go into your own user page and go to the reputation tab you can then see that in the last 5 days or so you've only had those two downvotes but no upvotes anywhere.
You received one upvote on that answer, and one downvote, for a net reputation gain of (10-2) 8 (as of this writing).
Users with at least 1,000 reputation can click on the vote and see the count of distinct up- and down-votes.
Use the reputation audit to see where you've gained, and lost, points recently.