Flags are usually decided on extremely quickly in chat, they hardly last longer than a minute at most. So what you're seeing are not some lingering flags you already voted on but that are still undecided, but the very aggressive caching of flags that have been dealt with. If the cache would be invalidated quickly after the flags are dealt with, the indicator would be reliable enough without having to change it to only showing unread flags.
There are also some edge cases that cause the flag count to remain cached for hours that significantly add to the unreliability of that indicator.
As it is now the indicator should be just removed entirely, the live indicator in the chat windows itself is enough.