I'm looking at my profile. One thread shows one post with a -1 and one with a +4. The +4 post is really mine, the -1 is someone else's that I downvoted and commented on. This doesn't seem right. It does show the -1 point for downvoting--it's like it's really displaying rep activity rather than posts.
That's exactly what it is doing. It's displaying your rep changes. You lost 1 rep for downvoting a question, that's a rep change, thus it's displayed. Don't stress, nobody can see that page except for you. And diamond mods. They can however see your public profile (see mine for an example)