IMHO, inability to search on site is one of the single biggest barriers.
Saying Google is the defacto UI is fine if you only ever do a single search and find the answer first try. (See point 7For reference, see Point #7 in the 'Nine Building Blocks' section of this article.)this 2009 article where Joel Spolsky speaks about this:
Google is UI: Stack Overflow was built around the assumption that people will go to Google which will send them to the right page. Each URL has the name of the question; each URL is permanent and clean, Metatags, sitemaps; anything and everything was done to ensure Stack Overflow's pages looked "reasonable to search engines."
But once you are already on S/OFU and you want to re-search (as in do another search) or find something else, are we really expected to go back to Google, do the new search, just to end up right back on the same site (S/OFU) we were just at?!
That's crapnot a good user experience.
And as Cletus mentioned in his answer above, I feel sorry for the poor n00bs who come here thinking it's awesome, ask a seemingly innocent question, and then get shouted/voted down for asking a dupe and not searching properly before they asked. And yes, I said 'shouted down', because despite what a lot of experienced users many think, that's what it'd feel like for a new user who's unfamiliar with the S/OFU culture.
I think improving the S/OFU search is really important, for heavy and new users alike.