I wouldn't remove a vote due to a subtle error. I'd leave a comment pointing out the error. If the answer is otherwise well written, it probably deserves a vote and a nudge to become a 100% perfect answer, or the author learns something from you. Either way, learning happens best when communication happens, votes are really just signals.
Still, there are times when people might argue that it is not an error at all, and the rationale behind their assertion. You might, in fact, learn two things from the answer.
Five minutes is really long enough if you've:
- Read the answer
- Tested any part of the answer that wasn't immediately obvious
- Viewed the answer in context with others.
I usually skim through answers, leaving comments (or perhaps down voting) answers that are just indisputably and blatantly wrong. After that, I decide which of the correct answers is the most comprehensive and well written. That one gets my vote.
You can vote as you read, but I prefer to spend a little more time.