Not every answer with just a link in it should be a comment. So long as the links are answering the question, I don't think flagging as "not an answer" is appropriate.
So I'd leave a comment instead of flagging. I usually do something alone the lines of
Your answer is helpful, but you can make it betteryou can make it better by including a summary or relevant portions of the pages you're linking to. This will also help your answer remain great even if the links you included break in the future.
If the user doesn't respond, I downvote since an answer with just links in it is not a good answer.