Personally, I don't think you should seek out a user's other answers. It doesn't seem like a natural occurrence to me - you only came across an answer to down-vote because you were looking into a user, not the question. If you happen upon a post that is absolutely terrible, then of course you should feel obligated to say so (though perhaps via a comment instead of a down-vote). Unless it's in a niche tag, though, or it's brand new, an answer bad enough for you to down-vote without any other context should probably have already been identified by other community members as such.
As LBT noted, if you do this for too many posts by the same person, they will be reversed and you'll likely be suspended.
You can certainly get a feel for a user's experience with the site by simply looking at the summary of their questions / answers, without reading the posts themselves. Again, unless they're in a niche tag or all brand new, there should already be some voting evidence from the community that is apparent right there. If you see a lot of 0s or -1s or simply not a lot of answers at all, then you can consider them one way vs. a lot of high up-votes, accepts, etc.
I don't think you need to gauge the quality of their actual answers (and down-vote them) to help you determine what type of comment you should leave. This is like trying to find out if your waiter's mother smokes (and yelling at her if she does!) before deciding what type of tip to leave.