Inspired by How did this person who has 1 rep comment on a question? and this comment, I think that adding a visual indication when a comment was actually an answer that was converted to comment by a moderator or by the system will prevent confusion.
Using the existing design of edited comment, what I have in mind is also an icon to the right of the comment timestamp:
With a tooltip like:
this comment has been converted from existing answer
Same goes for comments on such an answer, that also gets converted to comments on the question.
I don't think we need link to the deleted answer, just this simple indicator.