When you look at the flag dialog, we see the following:
it is not an answer
This was posted as an answer, but it does not attempt to answer the question. It should possibly be an edit, a comment, another question, or deleted altogether.
- Bold/italic emphasis is mine
Everything in ShaWizDowArd's answer is 100% solid advice, except for #3. Don't use the "other" option for this. If the answer should be deleted, that's precisely what the not an answer flag reason is for. If an answer isn't an answer, use the not an answer flag.
As for the reasoning behind this, keep in mind that only moderators can see "other" flags, whereas not an answer flags (and very low quality flags for questions) are visible to 10k and 20k+ users. What's more, 20k users can cast delete votes on these answers if they agree with the flag and the answer is negatively scored. This helps take the work off of the moderators, who at this point in time on Stack Overflow have at least 173 flags in the queue that are also visible to 10k users like me. There are likely many more "other" flags where the issue was so complex as to require a private message, and these are flags the community can't help with.
Use comments to kill 2 birds with one stone
As for moderators not understanding why you've flagged as not an answer, the solution is to leave a comment. As a community moderated site, we should get the ball rolling by letting users know what's wrong with their post and then explain what they need to do to fix that problem. Not only will this be seen by the user so he/she can fix the problem, but a moderator can also gain context from the comments, and that's only if three 20k users don't get there first.
Moreover, these comments educate the community. Users who are currently lurkers will see the comments and downvotes and hopefully avoid making the same mistakes themselves.
Lastly, consider that the problem you encountered is you flagged an answer that arguably didn't meet the criteria for a NAA flag as listed in the "What is an acceptable answer" table here. Thus, if you used "other", your flag may still be declined. Perhaps the table should be edited to account for cases where an answer is posted that doesn't fundamentally answer the question. Does that count as a wrong answer that should be downvoted but not flagged? Or is that simply considered an answer that doesn't answer the question which should be flagged? The answer of course in this case is to not flag it at all, since it attempts to answer the question and is a full, standalone answer, but I can see how this isn't clear to flaggers.