Voting is critical to new sites as the community works out what kinds of questions should be entertained. If we impede down voting, even just a little, it would probably result in the site taking much longer to graduate, or not graduating at all.
Down voting questions that you don't feel are a good fit is your way of saying "I'm a part of this community, and I don't think this will be good for us". We'd be foolish to get in the way when it comes to that.
Additionally, your pro-tem moderators (or SE employees at the beginning) desperately need this kind of signal so they can act on behalf of the majority, at least until a clear definition of scope can be established.
If you take that away, you end up with people that don't vote, and moderators that are reluctant to moderate as their binding vote now expresses fewer opinions, or perhaps only their own opinion.
For a new community, that would spell disaster.