Not every answer with just a link in it should be a comment. So long as the links are answering the question, I don't think flagging as "not an answer" is appropriate.

So I'd leave a comment instead of flagging. I usually do something alone the lines of 

> Your answer is helpful, but [you can make it better](http://meta.stackoverflow.com/q/8259) by including a summary or relevant portions of the pages you're linking to. This will also help your answer remain great even if the links you included break in the future.

If the user doesn't respond, I downvote since an answer with just links in it is not a good answer.