As a business model, you generally want to commoditize your complements ... not your own business.
For a for-profit business, open sourcing software makes sense if it drives sales of something else. What would releasing the SO engine as open-sources achieve from a business standpoint?
- Community contribution of development time? Possibly, but much of the cost of releasing software comes from testing and maintenance - not necessarily development.
- Having the community find and fix bugs? Can't see that for SO, it seems to be stable and the types of defects that are out there don't drive people away from using the site.
- Increased market visibility / marketing benefits? Possibly, since I suspect that sales of SO hosting will be driven more by word of mouth than advertising. However, having poorly managed clone sites could just as easily discourage customers - if they're bad enough.
Personally, I think it's hard to make a compelling business case for open sourcing something like SO - and I have yet to hear one.