+1 for the suggestion in your second paragraph, although IME the problem users typically don't show up until after the site has already established itself (i.e. there is a population of users available to annoy).

So during the formative stages of a site (when the users are, by definition, more dedicated than the casual ones that show up later) everyone is pretty much well-behaved, and the population is small enough so that the moderator can take care of any problems that do occur.