Here's an example of what moderators currently see when an answer is flagged for attention.
There are two small pieces of additional information that would speed up the processing of this kind of flag.
The date the question was posted. This could go right under the question title. I'd like to see the question date so I can see the lag between the question and the flagged answer. New answers on new questions seem a little more urgent to me than new answers on old questions, since the former is likely to be more active.
The fact that the person who asked the question also posted this answer. This could be accomplished with the same highlighting that's currently used in answers and comments when the OP responds.
If a post is flagged as "not an answer" I typically look at the date to see if it's fresh or not. If it's a really old post I usually just delete it from the flag page (using the handy "delete post" link). If it's new, I'll often go to the post and leave a comment saying that they should have posted a new question. This takes a little more time, but it's friendlier to new users.
If the "answer" was a follow-up question from the OP, I'd know that I probably need to go in and edit it into the original question. I at least need to go take a look. If it's a follow-up from someone else, I'd be more confident that I can safely delete it from the flag page without losing any information.