If the user is new (< 50 reputation) and posted a non-answer to a question that wasn't their own, then there is no sense in trying to teach them anything at all, since they don't have the comment privilege and thus can't even see the comment link.
Just flag it as "not an answer" and a moderator will convert it to a comment (or delete it).
If they have sufficient reputation to comment, or are "answering" their own question, then just point them to the above link and ask them to please post a comment next time and only use the answer box if they are directly answering the question.
Is there a UI problem? No, not really, I don't think so. There's a paradigm problem; 30+ years of BBSes and discussion forums have rotted people's brains and the very concept of a pure Q&A site is completely foreign to them. Treat it as a social problem and deal with it on a case-by-case basis; that's how real-life communities inculcate their community standards too.
There already is one type of protection, namely that answers detected by the system as "low quality" (which 90% of the time have the smell of a comment) will forward that user to the How to Answer page. At least, it does on the trilogy; maybe it's disabled on the newer Stack Exchanges.
Beyond that, I don't think it's productive for the system to browbeat users for their answers (or non-answers as the case may be). There's no foolproof way for the system to detect non-answers and the wording above the answer box (Your Answer) is pretty clear already. People won't read what you put in front of them, unfortunately, and sometimes it takes a couple of downvotes and/or flags to knock some sense into them.