The specialist silver badge should be awarded by having 400+ upvotes in a tag.
Some users already have this badge with less than 250 votes in this tag?
Does the number of accepted answers or anything else influence the calculation?
I think you're confusing "number of answers" with "number of votes".
Each answer can have more than one vote.
But remember that the tag badges ignore community wiki questions and answers.