Starting with a concrete example:
Let's say a question has two answers. Neither of them is very good, and would not otherwise warrant upvote, and neither is wrong or otherwise bad so does not warrant downvote either. Within this grey area of mediocre answers, one is still clearly better than the other, even if the difference may be just formatting or spelling/grammar or having some self-evident links to Wikipedia.
Should the relatively better one be upvoted just because it is better and should be higher under that question?
Should the relatively worse one be downvoted because it is kind of redundant, as the other answer is at least slightly better?
Or should both answers be evaluated on their own, on absolute merits, and in this example left without any votes?
Of course a real situation in a real question is rarely as clear cut as the example above, so the more general question is:
Should answer votes be used in an attempt to sort the answers under the question, or only to estimate the absolute merits of an answer?