Not every answer with just a link in it should be a comment. So long as the links are answering the question, I don't think flagging as "not an answer" is appropriate.

So I'd leave a comment instead of flagging. I usually do something alone the lines of 

> Your answer is helpful, but [you can
> make it
> better](http://meta.stackoverflow.com/q/8259)
> by including a summary or relevant
> portions of the pages you're linking
> to. This will also help your answer
> remain great even if the links you
> included break in the future.

If the user doesn't respond, I downvote since an answer with just links in it is not a good answer.