It happens pretty often
- up rated 1 or 2
- then some other people, for some reasons, just down vote that "best" answer
Isn't it annoying?
Ok, you may close that now :-)
It's not as annoying as seeing an incorrect answer get heavily up-voted because the voters didn't bother to read it critically or don't know enough about the topic to verify that it's correct.
And it's nowhere near as annoying as pointing out a problem with an answer, only to have the author whine about the down-votes he's now receiving instead of putting any effort into improving his work.
Not as long as they post a comment stating why they down voted it. Downvotes without justifying comments are always annoying, be it on an accepted answer or not.
And just because the OP accepted an answer, it need not be The Best answer.
Downvotes are supposed to be "annoying." If they weren't, people wouldn't pay any attention to them.
If you only received 1 or 2 upvotes before the downvote then I'm sorry to say that your answer is probably wrong, and simply got upvoted by a few people who didn't read the question or answer carefully enough and/or didn't really understand it. Because to be honest, that particular pattern of votes happens very seldom to many of us.
On the other hand, if I've posted a controversial answer to a subjective question and already received 10 or 20 upvotes, then I would expect to see a few downvotes in the mix. If you never manage to pick up a couple of haters then you're probably just boring everyone half to death.
Just because your answer got a measly 2 upvotes does not mean that it was a good answer. If you get a downvote under any circumstances then man up and check your answer for mistakes or poor writing. And if you're absolutely sure that your answer is perfect and nobody left a comment, then accept that the world isn't fair and that other people can be frivolous or even stupid.
Welcome to the internet. Please check your sensitivity at the door.
Are you perhaps confusing "best answer", "accepted answer" and "answer with the most upvotes"?
I assume you meant the latter, you got two upvotes giving you the lead, when some downvotes changed that. This may be because you have a mistake in your answer (in which case a comment would have been adequate) in which case your answer is definitely not the best answer.
But you could also be the victim of tactical downvoting, i.e. someone else wanted another answer to get more attention (when viewing sorted by votes).
Unless you point out the an answer that was treated the way you described, there's not much for anyone here to say but speculation ("link or it didn't happen")
No I'm not annoyed. Sometimes you get downvotes with no clear reason.
It's not worth to be annoyed for. The world has bigger problems.