I've been on SO for quite a while. It started as a very high-quality Q/A-site, and it still differs a lot from other forums I sometimes happen to stumble across.
However, the community has grown a lot since, and more and more not-so-skilled users start to contribute. I don't feel uncomfortable with that, but I noticed one major issue:
There is only one (numeric) means to measure the quality of a question or answer, it is the votes. Unfortunately there seems to be a natural law that the lower-quality a question is, the more upvotes it gets - because there are more users that understand the question and think it is not obvious. The votes then much more reflect the popularity of a question than the quality. I think that more qualified users tend to use the upvote feature more consciously than less qualified users do.
Downvoting is only allowed if the user has 125 points. Would it help to improve/maintain the Stack Overflow quality by restricting upvotes, too?
(One could even consider that you have to "pay" 1 point per upvote to get a similar effect as for downvotes, but this might be too radical.)
On my search for a duplicate question I found Should there be more incentive to upvote? which in my opinion supports this concern.