Does open sourcing Stack Overflow even make sense at all?
Yes, it makes sense. You get a horde of extremely competent and enthusiastic users as potentialcontributorspotential contributors to the SO code base. These users (me included) will contribute with new functionality, better unit tests, new designs, etc.
Are there "hybrid" models of open sourcing that could work?
It depends. Open sourcing the SO code base, but licensing such that it can't be hosted by anyone but you contradicts the FOSS spirit, but I guess it's possible. I can't think of a suitable existing license or a project sporting such a license at the moment.
Aren't there other companies pursuing open source and hosting businesses at the same time?
Indeed. WordPress.org is in direct competition with WordPress.com, TypePad is in competition with MovableType, etc. Both Automattic and SixApart seems to be doing just fine. Especially in the case of Automattic, I'd say they're doing a hell of a lot better after open sourcing WordPress than they did before.
Won't we be competing with open source versions of ourselves anyway in the long term?
Yep. I think the NRKbeta Doctrine applies just as much to source code and services as it does to copyrighted content. The only way to control your content is to be the best provider of it. If SO doesn't provide the community with the source code for SO-like services, someone else sooner or later will. Can and will SO risk loosinglosing the head-start to another service provider?
Could there be "enterprisey" closed source and "public" open source parts of the business?
Absolutely. See my answer to point 3. There's lots of other examples of service providers doing the same, but I think those examples weigh heavy enough (for now, at least).