Ben's handled the arguments for why votes to depose moderators are a problem, so let me counter the two items you list as negatives:
Those who do not leave the privilege for many years (remembering me of leaders of dark regimes who do not like to leave the throne).
Moderators are not kings, we're janitors and sometimes policemen. We don't rule communities, or impose policy. We're just there to handle the things that community moderation can't. We clean up spam, voting fraud, outright trash, abusive users, etc. We're volunteers who have been vetted by a crowd of our peers and entrusted with more powerful tools for dealing with these issues.
On newer sites, pro-tem and elected moderators can help steer community direction, but that generally lasts only until a site gets to a certain maturity. Even then, that's limited in effect.
There are a ton of technical ins-and-outs to being a moderator on one of these sites, from the moderator tools themselves to the layers on layers of community standards that have built up over time. Moderators that have been around for a while are likely to know all these intricacies and can help out newer moderators or those who might have missed a particular feature. There are also a lot of techniques that have been developed for tracking down sock puppets, trolls, etc. that can only be passed down in private between moderators.
It's actually difficult to keep moderators around for a significant duration (here's a chart illustrating that on Stack Overflow), so in practice you have a pretty natural rotation of moderators. Being faced with a constant stream of trash and abuse can get to us all, and we're all subject to life changes. The moderators I've seen who stick around the longest are the ones who keep an even keel in the face of the worst of the Internet.
Those who are moderators on many Stack Exchange websites
This type of moderators end up by rarely contributing to the communities as they have to deal with many things on the websites on which they are moderators.
I don't think I've seen this in any case. Stack Overflow is unique in the load it places on moderators, and all other sites don't have anywhere near those kind of demands. Moderators who work across multiple sites haven't been hurt by that, in my experience.
In fact, it can be an advantage where there is subject matter overlap between the two. I know that the fact that ChrisF is a moderator on both Stack Overflow and Software Engineering (formerly Programmers) has been a huge help when we get a bunch of questions flagged for migration to the latter. He knows the scope of both and can help us make those decisions quickly.
Someone with prior moderator experience on one site can often transfer that to a new site, so that's almost always been a positive to me when I see someone running for a moderator position. Many of the same arguments I made above about old-timers apply to this.
So in reality, both of the negatives you list are more likely to be positives for Stack Exchange moderators.