This idea is not as outrageous as it sounds. Sometimes I do get the vibe both that upvotes are a question of timing and that "rich get richer", sometimes independent of answer quality.
For example, a slight variation of the OP's suggestion is the standard procedure to review a scientific paper. You (and 2-4 other reviewers) get the paper (question) and are asked to write a review (an answer). Then the editors (or area chairs) collect all reviews and make a decision. You don't know the rep or who wrote the paper (question) and you do not know the rep or who the other reviewers (answerers) are. The reviewers are not allowed to work together, and do not get instant feed back on the factual accuracy of their review.
The system seems to be working reasonably well for science.
I do think that a timeout period, to "cool off" the popularity contest and during which you're forced to vote only evaluating the merits of the question and answer could be very helpful.
EDIT: This is one example after looking through one tag, for yesterday's activity:
Concatenating two strings iteratively in C++
In this case, I consider both answers at least equally good, and if pressed, I'd say the answer that came in second, is better, more thorough. BTW, as this will change during the discussion, when I linked it the first answer has 5 upvotes and the second 3 and the OP chose the first answer as correct.