First, a good-faith request: please don't take this too seriously. I've had this idea, ran it by a couple of people and decided it's not completely worthless. Just forgetting about it seemed a waste. I'm sure it's far too radical to be an actual realistic possibility on StackExchange. Still, here it goes - perhaps it'll spark some other, less radical yet realistic ideas.
To increase reward for posting many complex answers to obscure questions, and reduce reward for posting simple answers to basic but very popular questions. Not because the latter is not useful - absolutely not. But rather, because the system is currently disproportionately biased in favour of the latter.
In other words, I'd like to award the answerer on the basis of how close his/her answer is to being the best answer available, rather than on the basis of how popular it is.
Instead of awarding rep for every upvote or downvote of an answer, award rep according to the rank of the answer, as sorted by total votes.
The exact details can vary a lot, but one idea is that the rank awards a multiplier, while the weight of each vote depends on the total number of votes. Thus, the top answer might get a multiplier of 3, the second top answer 2, and all others 1. Moreover, a question whose total vote is +1 would get 10 base rep, while one with 3 upvotes would only get 10+9+8 = 27 rep. The maximum base rep would thus be 10+...+1 = 55, at which point any further upvotes are just "popular question" upvotes and effectively don't count.
Under this specific example (which is by no means the only way to implement the general idea of using rank), a top answer with +30 votes would get the answerer 3*55 = 165 rep - and that's it; never more. A top answer to an unpopular question with just +3 votes would still get the answerer 3*27 = 81 rep - only a factor of 2x out with a really popular top answer.
One idea for handling accepted answers is that its multiplier is the max of the rank-based multiplier and, say, 2 - so if it's top by votes already then it gains zero from being accepted, but if it's not top then it gains quite a lot, based on votes.
Obviously this is very sketchy. Among other things, it doesn't consider rep for questions at all, or rep loss for negative vote totals in any detal.
Please, if you're going to reply, don't nitpick on the very specific numbers I posted as an example - the example hasn't been carefully analysed or fine-tuned. Rather, concentrate on the idea of awarding answer rep on the basis of the rank of the answer.