If it looks like an answer (and not a follow-on question or a "me too" comment), but is simply written in another language, I might not flag these right away. What I'd do is leave a comment explaining that Stack Exchange sites are (for the time being) English-only, and that we'd greatly appreciate it if they'd reword their answer in English. Even better, if you were familiar with the language in question, edit the answer to translate it.
This at least explains to the answerer why there's a problem with their answer, and gives them a chance to correct it. I'd like to give them that opportunity, since they might not know about the English-only policy here.
If they don't respond to the comment in a timely manner, that's when I'd flag the answer with a custom flag explaining that they were asked to translate a non-English answer, but haven't. I have no problem with removing an answer at that point.