Absolutely not. The purpose of votes is to express what you think is a useful post and what you think is not a useful post. If you factor the current number of votes into your definition of usefulness, well, why not?
This isn't a new debate; Rosinante's answer hereRosinante's answer here summarizes the two voting models well:
- You compare the content to the current tally. If it's 'too high' you vote it down, and if it's 'too low' you vote it up. If it's 'just right' you leave it alone.
- You only look at the content. If [it's good], you add +1. If [it's bad], you add -1.
These models have been discussed on Meta since before it existed, but I suck at searching Meta, so if you want older discussions, go and search for yourself. I happen to mostly follow the second model myself, but there's nothing fundamentally wrong with the first model.
As long as your vote only depends on the post (and not, say, on the poster), you may vote as you want.
Several sites in the Stack Exchange network even have a policies that involve downvoting posts that match certain conditions (e.g. lacking references), but only if they are positively voted.