I've been recieving quite a bit of criticism over this question so I have edited it to hopefully reach a compromise that everyone can agree upon.
While I disagree that imposing a downvote limit from one person to another would negatively impact the SE voting system, I understand that imposing such a limit is not an answer that will be agreed upon and thus is not the proper solution.
For this reason I propose an alternate solution that does not impose any limits whatsoever. Instead of imposing limits, a simple notification system to let a user know when they are in danger of being flagged by the automated system for serial down/upvoting could be put in place.
Adding this alert would effectively discourage people who actually intend to "serial down/upvote." I'm sure most people who are serial voters are unaware that a system is in place to stop it, and just notifying them that this system exists would be enough to discourage many of them.
To address another point made:
This would also be very helpful to those who run into a user who frequently asks bad questions. As noted in a previous comment on Monica Cellio's response to this question, the automated system frequently fails when a user spams multiple bad questions and the downvotes are falsely detected as being "serial downvotes." (See here)
Alerting users that their votes may be reversed by the automated system if they do too many would relieve some of that problem. (Obviously it would not fix the problem, the system is inherently broken when dealing with situations like these, but a true solution to that issue should be left for another proposal).
The original question below is kept only for consistency and so all of the responses make sense. All future votes on this question should reflect the edit and not the original post.
I've asked numerous questions on Stack Overflow. Just recently I began to notice that almost every new question I ask always gets one downvote, regardless of the quality of the question.
Obviously I don't know who is downvoting my questions, or even if it is the same person each time; but it at least begs the question . . . Could there be someone out there purposely downvoting my questions because of a personal vendetta?
Not that this kind of rivalry happens frequently, but people have it out for each other in the comments section all the time so I wouldn't be surprised if some rivalries form.
I think it would be a good idea to limit how many times one user can downvote another user. (Obviously the limit would be reasonable, for example you may be only allowed to downvote another user's questions and/or responses 10 times in one month.)
Even if such a limit were imposed and you find that you reach this limit; if the user you wish to downvote truly has bad questions all the time, they will get enough downvotes from other users anyways.
Chances are if a user reaches this limit they are probably trying to abuse the voting system anyways. If a user frequently reaches this limit it would be a useful indication to look into them for potential serial downvoting.
To any naysayers to this question saying it would be a bad idea because it encourages users to ask bad questions... Think about it realistically. Would you frequently reach this limit..? I highly doubt it. And if you would frequently reach this limit, maybe you should stop for a minute and question your voting habits.