Is it appropriate to be voting on posts based on the poster?
No, but I don't see evidence that the behavior you mention is an instance of "voting on posts based on the poster". There exist some people who hold that any answer to low quality questions is harmful to the site (i.e. "not useful") and vote accordingly. We know those people exist because they sometimes say that's how they vote. I agree with those other folks who commented on the question here or posted an answer that such voting is not against the user but against the post.
But there's more. Oftentimes people who witness downvotes on answers to low quality questions assume that the downvote is due to the question being low quality when it is in fact not the case at all. I submit that low quality questions invite low quality answers. An unclear question, for instance, will elicit guesses. Some guesses can be very informed but very often someone will shoot from the hip and post a guess that cannot possibly be the correct answer. It is worth a downvote, not because answers to low quality questions should be downvoted as a rule, but because the answer was terrible.
Unclear questions can also receive "correct" answers that are still low quality answers. Let's say someone posts a question that says "my car says the pressure is low in my tires, what should I do?". A question like this should really contain more diagnosis information to be a good question. Did you check the tire pressure with a gauge? Are your tires keeping their pressure? If you inflate them is the warning going away? Etc. But someone answers "buy new tires". It is quite likely that this will fix the issue, even if the tires are not the problem. For instance if a sensor is failing, the mechanic will probably detect the problem once the new tires are installed. So this answer is not "incorrect" in the sense that "change your wipers" would be. However, there's a lot that should be done before someone buys new tires. If the issue is just a sensor, then only the sensor should be replaced. It is quite possible for an answer to be "correct" and yet be worthy of a downvote. I used a non-SO-specific example here but I see many answers of this sort on low quality questions posted on SO.
Similarly case is when people post their opinions, no matter how half-baked, to opinion-based questions. Chances are that I'm not going to be convinced by the half-baked opinion. Downvote it is.
Sometimes I see in the comments: "Why the downvote?" Someone else answers: "Oh, people are downvoting your answer because the question has close votes / downvotes, etc." Well, no. That's not it at all. The answer is a poor one and merits downvoting.