Another downside: drama. Consider this scenario: Two users are arguing with each other in comments. One of them now sees that some of his or her comments were flagged as rude, accuses the other one of doing that, and casts some retaliatory flags. The other user, who hadn't actually flagged anything, takes umbrage at the unfair accusations and returns fire. Soon the fight spills over onto other questions and answers, usually old ones. Moderators get a mess to clean up.
Why do I think this is a plausible concern? Because we already see it with downvotes, revenge downvotes, and accusatory comments. If we make flags more public than they are now, then it's reasonable to expect that a flagged user with a known sparring partner would behave as badly as a downvoted user with a known sparring partner does. (Maybe worse, because flags don't cost you any reputation, but flags have some auto-limits that would eventually help.)
A possible counter-argument to this issue is chat flags. Chat flags are visible (anonymously) to everybody on that chat server with 10k reputation. A single rude/abusive flag in chat can be seen by hundreds of people. (I don't know if people can see that their own messages have been flagged.) So how has this worked out for chat? I don't have good data here; sometimes rude messages are removed promptly, and other times people flood into a chat room in response to flags and suddenly the room is talking about flags -- "who flagged that and summoned people?" or similar sentiments. Sometimes that gets ugly, though I suspect these cases are a minority. Making chat flags broadly visible is probably a necessary evil -- flags are only delivered to people currently in chat, unlike flags on Q&A sites that are seen by moderators later.
Before making comment flags more visible than they are now, I'd want so see some more analysis of the drama issue. Looking at chat could be helpful, but that's hard for most of us to do.