Not everyone upvotes questions they answer.
On quiet questions, it's easy to tell how the user who answers has (not) voted on the question. Do your reactions to an answer vary when you can tell that the answerer has (not) upvoted your question? If they do, are the reasons intrinsic in how answers are written, or in a "prejudice" on the answerer's intentions?
In my observations, people are more likely to "return" an upvote to my answer if I have upvoted their question.