It's a simple idea, and I think it's something that is useful. It can be a lot of extra work, though, keep in mind.
Sometimes people trip up. They may otherwise make a lot of great flags, but then make one mistake that we need to dismiss as invalid. This is fairly fine, and really doesn't need to be reported or dwelled on. It's not particularly condemning if you make a bad flag once or twice, as long as you don't develop a habit or get comfortable doing it.
Also keep in mind that if possible, a moderator may comment on the post to address the course of action if doing so would not betray the trust in privacy of the flag. This can handle a lot of one-time mistakes that a new flagger may run into.
This is what makes your proposal tiresome to implement, take into account that we may yet dismiss such one-time flags as valid. It may be because we saw something else as a problem, or maybe we think that others may make the same misconception. Whatever the scenario is, the flag led to action, regardless of whether it was the action intended by the flagger. So, to properly know what is good flagging, haha, not even the exactness of valid/invalid dismissal will give you an accurate reading.
To that end, proper feedback would entail writing a response for every single flag that comes in. Addressing all but the most basic of flags is rarely black-and-white. Considering I see some Stack Overflow mods clamoring at the thought of automated comments for deletion or other warnings... well, I can't speak for them in the end. But they have a much greater volume of flags than any of the other sites have to deal with. That's a lot of writing back to be done.
Feedback remains important, but I think the real important thing is to notice when someone is actually making some kind of trend of bad flagging. Words will tell you what's going on a lot better than the dismissal stats will, and I think the words are more valuable and more likely to get accurate readings when they're analyzing something other than a one-time trip.
