Actually, the main psychological phenomenon underlying Stack Overflow (and numerous other sites with similar reputation/karma/badge/reward systems) is something that's been discussed on meta before: Intermittent Variable Reward. Although I think the "correct" term (or at least the one more commonly used) is simply intermittent reinforcement. Intermittent implies a schedule of some kind, and the schedule here is called Variable Ratio. Generally the whole thing is colloquially called random reinforcement, as it's a well-known technique.
Positive and negative reinforcement are obviously pretty basic concepts. When subjects respond correctly to a cue, you give them something they want. When they refuse or do something bad, you punish them either directly or indirectly (by taking away something they want).
What a lot of people don't realize is that with positive reinforcement, the conditioning doesn't really "stick" if you always give the reward. They'll keep doing what you want as long as you keep giving the reward (and as long as they still want it), but as soon as you take the reward away, they will stop. The only reason to perform the behaviour is to get the reward, so in their mind, there's no longer any reason to continue to cooperate. The technical term for this is extinction.
Switching to a variable ratio of reinforcement - meaning you give 1 "unit" of reward for every X responses, where X changes randomly after each reward - is one of the most effective forms of conditioning and almost universally used by animal trainers. It is highly resistant to extinction; because the subject never knows exactly when the next reward will be, they will continue to work for it almost indefinitely.
The result of this, particularly in humans, is addiction. People make up for the uncertainty of a reward with increased activity, hoping that it will get them "enough" of the reward.
Stack Overflow (and similarly-designed sites) stumbled upon this accidentally, or at least it would appear that way. Individually, the system of reinforcement appears to be continuous; you see a good answer, you upvote, you see a bad answer, you downvote. But when you look at the big picture, which includes:
- Different users being online at different times;
- Other questions competing for front-page space;
- Differences in subject matter and technical knowledge among readers/voters;
- Vote limits, voting incentives (electorate etc.) and other artificial stimuli;
...and so on - then what it all amounts to is one massive system of random reinforcement. Posting what you think is a good answer means you might get some reward for it. The emergent schedule is actually pretty similar to what's used in ordinary conditioning, usually dishing out a reward for every 1-10 responses. And of course, the size of the reward also varies; setting artificial, relatively difficult-to-reach targets like reputation caps help to cement the sense of proportion.
Perhaps going off on a tangent here, but I think it's interesting to add, if you switch to a reinforcement schedule that's random based on time and is not tied in any way whatsoever to the behaviour itself, you'll often see the phenomenon of superstition and cult-like behaviour. Think of rain dances, ritual sacrifices, even quack medicine - humans become obsessed with these because they sometimes seem to "work." Unlike animals, humans are able to form far more complex associations over much longer periods of time, and as a result tend to latch onto what they think is a pattern and interpret any future results as VI reinforcement. They're doing it right, they just have to keep at it a little longer!
I suppose we could have another long and tedious discussion about what makes the intangible reward of reputation/badges so attractive to members, but the truth is, it doesn't really matter what the reward is as long as it's measurable. Something as basic as a compliment works just fine. It's why people obsessively play RPGs; every now and then they get a level-up or something similar which communicates the concept, "you're awesome." Reputation and badges are really no different - they're little compliments, randomly given for some definition of positive behaviour.
There are many other psychological concepts you may witness within Stack Overflow - but most assuredly the one it takes advantage of is random reinforcement.
