Generally I will only make a comment when downvoting if that comment is actionable.
If an answer is inane or appears to be intentionally unhelpful, or just beside the point, I won't bother leaving a comment. At some point I'd just get tired of writing "don't be an ass", "don't be rude", "don't be an idiot" comments.
If an answer appears to be intended to be helpful but has some flaw, I will be more likely to add a comment. The answerer may have misinterpreted the question, or they may have made some error in coding, or they may not be aware of the deficiencies of their design, so a comment is worthwhile because it gives them feedback and it gives others who read the answer useful information (on whether to use that answer, for example) as well.