When I am writing an answer, especially a detailed one, it's useful to be able to refer to the question to make sure that I am answering everything and doing it in a logical order. Because of this need to refer back to the question, I have to scroll up and down the entire page which takes a while when there are lots of existing answers. This can also be annoying when a lot of the answers I am scrolling past are ones that are greyed out because they did not answer the question.
The crux of the issue is that Stack Exchange is a question-and-answer site, but your answer and the question you're writing an answer to are placed on opposite ends of the page.
The current page structure looks like this:
One (and my preferred) solution is to have a "Write an answer..." button underneath the question which when clicked on expands to reveal the answer box directly below the question. That way the question and your answer are side-by-side. You could still have another "Write an answer..." button at the very bottom of the page which jumps to the question and does the same thing.
Another solution could be to add buttons (which only appear when you have entered something in the answer box) to quickly jump between the question and your answer: