I'm a moderator on a small site, and I've noticed a pattern in my voting. When a post is VLQ or off-topic, it's easy to comment explaining why and close/delete, and I don't feel bad about that. It's usually quite obvious on LQ posts, and OT is similarly straightforward most of the time. Sometimes others have already voted/commented as well.
Where I encounter problems is when I come across an answer that is factually incorrect. It is very important to downvote incorrect answers so that future visitors don't think they're right, especially on small sites where there's less voting, and if you don't downvote nobody might. A zero score on a solitary or accepted answer (on a site with low voting) seems positive; there's certainly no sign to the OP or future visitors that it's wrong. So downvoting is crucial in these cases.
It's also important in these cases to comment explaining why the answer is wrong. This is important for two reasons: 1) the author could then edit the answer to remove the incorrect statement(s), and 2) future readers can see an explanation of what's incorrect, helping them further to not internalize wrong information.
Here's where the problem comes in: I feel perfectly fine doing this on sites where I'm not a moderator. I understand it's the best thing to do for everyone involved (even if sometimes people who are downvoted don't feel what way), and I have no qualms about doing so. But as a moderator, I feel bad casting the downvote and leaving the comment shortly after, clearly tying the vote to myself. (It's pretty easy for the user to tell; they can see when the comment was posted and check their rep history. When you don't get a lot of downvotes there's always the possibility it's a coincidence, but... the likelihood is pretty high.)
The reason I feel bad is because of the diamond. I don't worry when the user is fairly new, because I think they'll react to my comment/downvote the same way as they would anyone else's. But when you have an established user who you've interacted with before, and you're (politely, but still) poking holes in their answers and associating yourself with a downvote, I feel like my criticism carries more "weight" than others, and that I might be making the user feel bad. I vaguely recall having felt that way in the user's place before, though I don't recall when and it might not have been on SE... But I definitely remember the feeling of "Wow, this was bad enough to get a moderator's attention? I must really suck."
And of course that's not the feeling I'm trying to get across, and my comments are always kind and matter-of-fact, but I still feel like I might be putting undue pressure on users that might not be there in absence of the shiny diamond. And so I'm not sure if I'm doing more harm than good; I know that the downvotes and comments are helpful, but I don't know if I'm negating that effect by associating myself with it. I know that I wasn't successful in inspiring change the last time I tried this, and that in the past I've had users delete the answers within minutes of my comment. I didn't mean for them to delete it! Editing to fix is of course always the desired result.
So... I don't know. Am I making a mountain out of a molehill? Is this problem entirely in my head? I feel like it's sort of similar to moderators having binding votes, though; if you're going to make a decision to do something, it's going to be seen through moderator-lenses. It's not just your opinion anymore, you're representing moderation. And so I feel like maybe that's making my comments have a different effect than I intend. So what should I do? The options seem to be: 1) keep doing what I'm doing, 2) keep commenting but stop the downvotes. I see pros and cons to both.
What say you to my giant wall of text, meta? (The sad part is I was really trying to keep this short.)