Yes, some people will systematically downvote answers to poor questions, irrespective of the quality of the answers.
However, the fact is that poor questions elicit poor answers. It is perfectly possible for a poor question to get a bunch of answers which are all poor in their own right. The fact that all answers on a poor question have been downvoted is not sufficient proof that someone has been downvoting answers merely because of the question. The answers may be bad in and of themselves. I particularly recall a good regex question on SO that got a total of four answers that were all incorrect. If all answers on a good question can be incorrect, it is all the more likely on a bad question.
I often see guess answers posted on questions that underspecify the problem. A guess can be informed or it can be a shot in the dark. Sometimes there is not enough information to tell what the real issue faced by the OP is but there is enough information to know that some of the guess answers cannot possibly be the solution. Of course the answerer is baffled when the answer is downvoted but it is not proof that the downvote is due to anything else than the quality of the answer. If you guess an answer, then unless you truly know the topic involved by the question inside and out, there's a high chance that your guess is wrong.
There's also the case of opinion-based questions being posted in venues where opinion-based questions are not appropriate. For instance, the main SO site. (By opposition to SO's Meta, which has more tolerance for opinions.) Some think that if they merely parrot a popular opinion, surely their answer is worthy of an upvote. Voters may set the bar higher than this. As I see it, the answerer should at least support their opinion with substantial reasons supporting. ("Do X because [some known name] says you should do X" is not a solid reason.) An opinion that does not do this is worthy of a downvote.
My point is yes it does happen that people downvote answers due to the quality of the question, but there are cases where it may look like this is happening, when in fact it is not.
I prefer not to answer low quality question as I feel it is a waste of time. It does not mean I never answer such questions but when I do, I make sure my answer is an informed one (no shots in the dark) and I'm willing to live with the downvotes of those who cannot tolerate answers to such questions. (I did get a few.)