Your question wandered across a few boundaries, so I'll attempt to disambiguate, then answer.
If I design a Stackoverflow shirt,
what would be the legal extent to
which I could profit from my work?
(edited to remove some legal jargon I had no right to have written in the first place)
Assuming that you had not announced your intention to do so ahead of time, the words (or compound word) "Stack Overflow" are just words. So if you create a shirt that says "stack overflow" anywhere on it and it has nothing to do with (or reference) this site then it is fine.
On the other hand, if you want to use the Stack Overflow typeface, the Stack Overflow logo, the Stack Overflow url, your Stack Overflow flair, or anything derived DIRECTLY, or inspired DIRECTLY by this website, then you should talk to a LAWYER. I don't mean to yell at you, I'm just trying to emphasize the lack of legal expertise that exists in the metaSO and SO communities.
What if a user wrote an app that used SO as an engine for material, could he sell licenses to the program?
I don't know what you mean by using "SO as an engine". If you are using the SO datadump as your source material, I'm fairly sure that's illegal, (again, see a lawyer). And that also answers the licensing question.
people who do a great deal of work, and happen to use bits of stackoverflow in their projects
If you are talking about incorporating the answers you find here into your code, I believe Jeff put something in a blog-post about the license he wanted to use. Basically you could re-use anything here as long as you didn't CLAIM it was your own. Proprietary/non-proprietary, it was all good as long as you didn't in some way claim that you yourself wrote it and not the original author.
As for incorporating the site into your application in some direct manner, then again, your lawyers would have to work that out with their lawyers. And your people with their people. I imagine it would involve a lot of expense'd lunches. If that is the case, I'd like to come too.