It isn't rare when you downvote someone and explain your reasons that he returns a downvote on a totally unrelated question, most likely your latest or highest-scoring.
Yes, this is not a big deal (at least until it becomes serial downvoting the target user), but I think something simple can be done to get rid of at least half of those.
The suggestion is to disable downvoting for 5/10 minutes after you have been downvoted or after you've logged in after being downvoted. This will cool off the desire to respond to the downvote.
Update: this will rarely have "innocent victims", since the downvotes are not that commons. And if you really want to downvote someone based on arguments, well, wait 10 minutes.