Some comments or answers ask users to alter their behavior on Stack Exchange. For example, critics may ask the questioner to modify their question, comment, or answer to fit some particular characteristics the critic has in mind. I understand there are the obvious feedback mechanisms like downvoting and flagging, but is it ever appropriate to act as "Stack Exchange police" and directly ask users in comments or answers to behave differently on Stack Exchange? If so, when, why, and how?
In general, yes. That's called "Community moderation" . Telling users that so-and-so is discouraged/against the rules is a good thing. We work differently from a forum, and this is what comments are for--to help others improve their post(s).
The cases you're talking about may be different, it looks like you're talking about people who are being over-nitpicky. Examples please?
If you see something you don't quite like in a comment, flag it. If it gets declined, the decline reason may be informative (you can see a list of your flags if you click the number next to "helpful flags" on your profile)
Stack Exchange is actually pretty complicated, and we have pretty high quality standards. SE really doesn't work without this sort of community moderation. The FAQ can't explain everything nor can moderators; there are too many users and too many things to explain everything to everyone unless the whole community is involved.
When a user doesn't quite "get" how something works or should works on Stack Exchange, you help them and the community by nudging them in the right direction in a polite and civil manner. It's not "policing" or "changing their behavior" so much as it is teaching them.
If users are asking questions that are out of line with the StackExchange model (such as gimme teh codez questions or broad, subjective, discussion-oriented questions), they are going to get a lot of downvotes, closures, and deleted questions.
If left unchecked, this will lead to clutter of things the rest of us don't want, and the user will likely get question-banned.
It is best to let users know what they are doing wrong, why that particular is not a good fit for the site, and what they can do to improve. This leads to less garbage content, more acceptable content, and the user not getting question-banned.
As for the "when, why, and how" (or at least the when and how, since I believe I've covered the why), comment on their post and politely explain the reason their content is not appropriate, and (if possible) suggest how it might be fixed. Alternately, depending on your reason for needing to direct their behavior, you could directly edit the post, and then leave a comment (or use the edit summary) to explain why you made the edit.
Lasty, the when. Whenever you see content in need of improvement. Seriously. If it needs a fix, or a user needs some advice, step in and lend a hand before they get barraged with downvotes or question-banned..