Maybe there's another way to look at this. Maybe the "problem" isn't the upvoting itself, maybe it's the idea that all upvotes are identical.
If reputation is a measure of the community's trust in someone, then the value of an endorsement of an answer should, logically, scale with reputation. Or to put it more simply, Jon Skeet upvoting someone has more meaning than if I do.
To take this to a more fully-fleshed system, you could argue that an upvote from someone with approximately the same rep as you is worth 10 points, as it is today. An upvote from someone with 1 rep might be worth a single point, while a Jon Skeet upvote could be worth significantly more (20? 50? 100? I hear that an upvote from Jon Skeet actually changes the correct answer to match what was posted...)
This may not change the actual upvoting, but may change the impact of it - presumably the drive-bys would be from more casual people, and so would have little impact on a reasonably-repped user. However, even a single upvote from a well-respected user could have significant impact.
Similarly, upvoting something is an endorsement of the answer. Perhaps future votes on that answer should reflect on the rep of the up/down-voter? So if you upvote something that then proceeds to get 100 downvotes, including the Mighty Downvote of Jon Skeet, then that's probably a sign of a lack of judgement on your part, and a rep hit may be appropriate. This may also encourage people to annotate answers in comments rather than writing a new, but basically identical answer that just modifies one small bit. This idea is (obviously) ripe with abuse potential, and would need to be thought out more fully.