Once I got home, I tried just going to an answer and typed in "not an answer" in the custom flag box. Viewing it directly from the question and in the flags queue, it shows up as a normal not an answer flag rather than a custom flag. Oddly, though, typing in "very low quality" does not exhibit the same behavior.
Likely the system just looks for flags which have the text "not an answer" as the flag reason and doesn't pay attention to anything else about the flag. You could consider this a bug, I guess. But I honestly don't think it makes any difference. Whether or not other users see your basic "not an answer" flag, custom or not, does not really change the end result. A moderator still has to see it in order for it to be declined, and sending it to 10k users still allows them to vote to delete it if it's negatively scored.
Bottom line - the way you flagged it, it was still a "not an answer" flag even if you typed it manually in the custom flag box, so there's not any problem with the system treating it as such. I'd much rather the system check the text you entered against previous flags and reject it if it's the same, but then again, people can just as easily find a way around that by flagging it as "not an answer!"